Source-linked AI summary
CM2: Multimodal Cultural Reasoning via an Integrated Multi-Agent Framework
Qi Li, Zhaojie Kang, Yingjie He, Zheng Lin, Hao Zhang, Guangxin Wu, Yan Gong, Rong Fu, Jianyuan Ni
TL;DR
Multimodal cultural reasoning remains less explored than STEM-style vertical deduction, especially where interpretation requires heterogeneous evidence and ambiguity resolution. CM2 addresses this gap with an integrated multi-agent framework and CM2D, reporting consistent gains over established reasoning paradigms across multiple MLLM backbones. Its analyses support contributions from the individual modules and genuine cross-modal arbitration.
Problem
Cultural reasoning is underexplored and often weakly evaluated despite requiring heterogeneous evidence integration, ambiguity handling, and context-sensitive interpretation.
Method
CM2 combines multimodal perception, retrieval, external verification, gated conflict arbitration, and reward-driven refinement, evaluated with CM2D.
Results
CM2 improves consistently over CoT and typical reasoning paradigms across multiple backbone MLLMs, with ablations and conflict analyses supporting module contributions and cross-modal arbitration.
Takeaways & Limitations
CM2D and CM2 provide a framework and benchmark for evaluating horizontal, context-sensitive cultural reasoning beyond surface recognition.
Abstract
from arXiv · showhide
Multimodal Large Language Models (MLLMs) have shown remarkable success in STEM domains, where progress is often driven by vertical, step-by-step deduction under relatively stable symbol systems. Their horizontal, interdisciplinary cultural reasoning, however, remains underexplored.We propose CM2, a multi-agent framework grounded in the cognitive pathway of human cultural interpretation. CM2 integrates multimodal perception, retrieval-augmented generation, networked reasoning, gated fusion, and reward-driven feedback.Experiments on CM2D across multiple MLLM backbones show consistent gains over CoT and typical reasoning paradigms; ablations validate each module's contribution, and conflict analyses confirm genuine cross-modal arbitration.
1 Introduction
CM2 addresses the limited evaluation and weak grounding of multimodal cultural reasoning by combining complementary evidence, conflict arbitration, and iterative refinement. It is accompanied by CM2D, a curated dataset designed to test horizontal, context-sensitive interpretation.
- Motivation: Cultural reasoning requires integrating heterogeneous evidence across modalities and disciplines while resolving ambiguity among plausible interpretations.The same motif can carry different meanings across periods, styles, or traditions, making evidence attribution and conflict resolution central.
- Motivation: Current cultural evaluations often simplify interpretation into QA or recognition decisions, leaving interdisciplinary integration and ambiguity handling under-tested.These settings are associated with fluent but weakly grounded explanations, superficial visual reliance, and unsupported culturally plausible narratives.
- Framework: CM2 integrates multimodal perception, retrieval-augmented generation, networked reasoning, gated fusion and synthesis, and reward-driven feedback for structured cultural reasoning.The framework generates complementary evidence, arbitrates cross-modal conflicts, and iteratively refines outputs under cultural reasoning criteria.
- Contributions: CM2 explicitly targets horizontal cultural reasoning through parallel evidence generation, conflict-aware fusion, and feedback-driven refinement.Its five-module framework is presented as the paper’s core contribution.
- Evaluation: CM2 shows consistent gains over CoT and typical reasoning baselines across multiple backbone MLLMs, with ablations and conflict analyses supporting genuine cross-modal arbitration.The reported evaluation claims cover both performance improvements and the role of the framework’s modules.
2 Related Work
Prior multimodal progress has concentrated on vertical deduction under fixed symbolic systems, while cultural interpretation requires heterogeneous evidence integration, ambiguity handling, and context-sensitive reasoning. Existing cultural evaluations often reduce these demands to direct QA or recognition-like decisions.
- Existing paradigms: STEM-oriented multimodal benchmarks primarily test vertical, step-by-step deduction under stable symbolic rules and well-defined objectives.These benchmarks have driven progress in visual encoders, scaling, and prompting.
- Cultural reasoning: Cultural interpretation instead requires integrating heterogeneous cues, handling ambiguity, and reasoning under context shifts.This motivates frameworks beyond generic prompting or retrieval.
- Evaluation gap: Existing cultural evaluations often simplify tasks into direct QA or recognition-like decisions, potentially missing interpretive nuance.HSSBench characterizes related failures as shallow symbolic associations, ungrounded narratives, and weak evidence reconciliation.
3 Method
CM2 maps a human-aesthetic-interpretation pathway onto multimodal evidence generation, retrieval, verification, gated arbitration, and reward-driven refinement. The resulting pipeline produces a final prediction from perceptual, retrieved, and externally verified evidence.
- Design rationale: CM2 characterizes cultural interpretation as horizontal reasoning and maps perceptual analysis, schema activation, classification, conflict arbitration, and evaluation to dedicated modules.The framework treats interpretation as a dynamic cycle rather than linear deduction.
- Task formulation: The task formalization represents an image, query, candidate options, and ground-truth answer, then fuses three evidence streams into a final prediction.The streams are perceptual Emp, retrieved Erag, and verified external Enr.
- Multimodal Perception (MP): Multimodal perception extracts culturally salient visual descriptions, structured questions about motif, period, and region, and a task-relevant summary.These components form Emp = {Dmp, QAmp, Smp}.
- Retrieval-Augmented Generation (RAG): RAG retrieves culturally relevant precedents by combining visual and textual cosine similarities with α=0.5, then concatenates the top-K=5 entries as downstream context.The curated knowledge base contains 3,715 image–text entries.
- Networked Reasoning (NR): Networked reasoning converts unresolved claims into search queries and accumulates top-r=5 snippets for iterative external verification.The loop runs for at most Tmax=3 iterations before summarizing accumulated evidence into a verification report.
- Gated Fusion & Synthesis (GFS): Gated fusion scores each evidence source on relevance, factual accuracy, and option discrimination, then softmax-weights source embeddings before synthesis.The synthesis generates a fused answer and rationale, allowing culturally grounded evidence to override superficial cues when conflicts arise.
- Reward-Driven Feedback (RDF): Reward-driven feedback scores evidence integration, visual grounding, logic, option differentiation, conclusion strength, and completeness, refining outputs when the average score is below τ=0.6.After at most two feedback rounds, the answer with the highest score is selected.
4 Dataset
CM2D is a curated multimodal dataset designed to evaluate horizontal, context-sensitive cultural reasoning beyond surface recognition. It combines diverse taxonomies, expert–agent construction, and checks that require cross-modal grounding.
- Dataset Overview: CM2D contains 4,157 image–question pairs: 442 expert-verified test samples and 3,715 retrieval entries.Each instance is a 4-way multiple-choice question with one correct answer and three culturally plausible distractors.
- Dataset Overview: The dataset covers symbolism, style, period or region attribution, medium conventions, and contextual interpretation.
- Taxonomy and Visual Diversity: CM2D organizes cultural materials into four coarse categories and thirteen fine-grained taxonomies based principally on cultural function and social medium.Instances receive the most salient label despite intersectionality among humanities categories.
- Taxonomy and Visual Diversity: The four categories span visual and fine arts; material culture and applied arts; spiritual and Indigenous traditions; and cultural practices and performance.
- Hybrid Construction Pipeline: A hybrid expert–agent pipeline aggregates authoritative materials and drafts semantic content from expert-provided seeds.The pipeline is described as operating in three phases, beginning with multimodal aggregation and continuing through semantic drafting.
- Quality Assurance: Counterfactual text-only and image-only screening revises or discards trivially solvable items so decisive evidence requires cross-modal grounding.A final expert pass confirms the multimodal dependency of retained questions.
5 Experiments
CM2 improves cultural-reasoning accuracy across multiple MLLM backbones and reasoning baselines on CM2D. Ablations and conflict-focused analyses attribute the gains to complementary modules and gated cross-modal arbitration.
- 5.1 Main Results: CM2 yields consistent accuracy improvements across all evaluated backbones, including llava-onevision-7b and GPT4.1-mini.Accuracy rises from 23.53% to 50.68% for llava-onevision-7b and from 41.63% to 55.20% for GPT4.1-mini.
- 5.1 Main Results: CM2 maintains a 4.75–10.86pp margin over the best baseline across backbones, narrowing to 0.90pp on GPT4.1-mini.
- 5.2 Ablation Studies: Every ablated module provides complementary value: MP adds stylistic cues, RAG adds precedent grounding, NR reduces unsupported narratives, GFS handles disagreement, and RDF targets correction.
- 5.3 Fine-Grained Evaluation under Evidence Conflict: On the Consensus Subset, representing 45.7%–46.2% of samples, CM2 leads both representative backbones by +4.9–11.4pp over the best baseline.This result is reported for cases where all three evidence sources agree.
- 5.3 Fine-Grained Evaluation under Evidence Conflict: On the Conflict Subset, CM2 reaches 50.0% and 53.4%, exceeding CoT by +18.3–26.1pp and the best competing method by +7.2–9.9pp.The subset comprises 54.3% of Qwen2.5-VL-7B-Instruct samples and 53.8% of llava-onevision-7b samples.
- 5.3 Fine-Grained Evaluation under Evidence Conflict: A qualitative lotus case shows baselines overfit superficial visual cues or stereotypes, whereas CM2 integrates perceptual style with retrieved cultural knowledge to recover resilience and vitality.The displayed reasoning texts are human-edited summaries rather than verbatim model outputs.
6 Conclusion
The paper presents CM2 and CM2D for context-sensitive horizontal cultural reasoning. Across multiple backbones, CM2 improves over CoT and typical reasoning paradigms, while ablations and conflict analyses support its modular contributions and cross-modal arbitration.
- 6 Conclusion: CM2 combines parallel evidence generation, retrieval grounding, external verification, gated conflict arbitration, and reward-driven refinement.
- 6 Conclusion: CM2D is a curated multimodal dataset emphasizing context-sensitive interpretation with multimodal dependency.
- 6 Conclusion: Across multiple backbones, CM2 improves over CoT and typical reasoning paradigms, while ablations confirm each module’s contribution.
- 6 Conclusion: Conflict analyses validate genuine cross-modal arbitration.