PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Modal Generative Fuzzy System: Fuzzy Inference Guided Large Model Interactive Question Answering Framework

Hailong Yang, Jianqi Wang, Guanjin Wang, Zhaohong Deng

arXiv:2608.14584v1cs.CLcs.AI

TL;DR

Multimodal QA remains limited by modality bias, cross-domain uncertainty, and shallow reasoning. MMGFS combines collaborative multimodal rumination with fuzzy rules and multi-hop inference, outperforming existing methods across open-domain and domain-specific benchmarks.

  • Problem

    Existing multimodal QA methods lack effective modality collaboration, struggle with multi-domain uncertainty, and often provide shallow semantic reasoning.

  • Method

    MMGFS aligns multimodal inputs in natural language, refines them collaboratively, and applies fuzzy domain rules with multi-hop inference for uncertainty-aware reasoning.

  • Results

    MMGFS consistently improves performance across open-domain and domain-specific tasks, with ablations showing gains from shared information, rumination, and fuzzy reasoning.

  • Takeaways & Limitations

    MMGFS supports more robust, generalized, and cognitively consistent multimodal question answering under uncertainty.

  • Takeaways & Limitations

    MMGFS supports a limited range of modalities and remains sensitive to semantic conflicts in multimodal inputs, potentially affecting reasoning reliability.

Abstract

from arXiv · show

In Multimodal Question Answering (MQA), models are required to jointly encode and integrate heterogeneous information from multiple modalities, including text, images, and speech, to perform complex semantic reasoning and decision making. Despite recent advances, existing approaches, including traditional deep learning models and Large Models (LMs) or prompt-based frameworks, continue to face several critical challenges. First, modality bias arises from discrepancies in feature distributions across different modalities, which limits effective cross modal collaborative understanding. Second, many questions require knowledge drawn from multiple domains, introducing significant uncertainty. Third, current methods often rely on shallow semantic matching, resulting in limited reasoning depth an reduced interpretability. To address these issues, inspired by the traditional fuzzy system (FS) framework, we propose a fuzzy-inference-guided multimodal generative architecture termed the Multi-Modal Generative Fuzzy System (MMGFS). The main contributions of MMGFS are two folds. First, it alleviates modality bias through a multimodal collaborative rumination mechanism. Second, it introduces fuzzy rules and a multi-hop inference mechanism to support cross-domain knowledge fusion and hierarchical reasoning, thereby strengthening uncertainty modelling and deepening semantic understanding. We conduct comprehensive evaluations on open-domain question answering datasets, including MultimodalQA and WebQA, as well as domain-specific benchmarks, including BioMol-VQA and EHRxQA. Experimental results demonstrate that MMGFS consistently outperforms existing methods across multiple datasets. It effectively mitigates modality bias and question uncertainty while achieving superior performance in answer accuracy, consistency, and generalization.

I. INTRODUCTION

The introduction frames MQA as multimodal joint understanding and reasoning, identifies modality alignment, multi-domain uncertainty, and shallow inference as key limitations, and presents MMGFS as a fuzzy-reasoning-guided LM framework addressing them. Its contributions include collaborative multimodal rumination and fuzzy-rule-driven hierarchical reasoning for uncertainty-aware knowledge fusion.

  • MQA background: MQA jointly processes text, images, and speech to support cross-modal reasoning and answer generation from integrated understanding.The task requires extracting modality-specific features, learning semantic correspondences, and enabling information fusion and multi-step reasoning.
  • Open challenges: Existing MQA methods struggle with heterogeneous feature alignment, multi-domain uncertainty, and shallow reasoning that limits long-chain inference and interpretability.Cross-modal attention can miss multi-entity and multi-relation interactions, while pattern matching performs poorly with implicit inference and external knowledge integration.
  • Proposed framework: MMGFS is a fuzzy-reasoning-guided LM interactive architecture that aligns multimodal inputs into natural-language representations and mitigates modality bias through collaborative rumination.It also performs domain fuzzification and rule fusion over questions and uses fuzzy reasoning for multimodal question answering.
  • Core mechanisms: The multimodal collaborative rumination mechanism dynamically aligns and iteratively refines text, images, sequences, and tables at the natural-language semantic level.The mechanism is designed to alleviate modality bias and enable cross-modal collaboration and knowledge fusion.
  • Core mechanisms: MMGFS introduces fuzzy-rule-driven hierarchical knowledge fusion and dynamic reasoning to improve uncertainty handling and deep semantic reasoning across multi-domain and multimodal tasks.The work presents MMGFS as an extension of classical fuzzy architectures to MQA and related multimodal reasoning tasks.

II. RELATED WORK · A. Deep Models for Multimodal Question Answering · B. Large-Model Interactive Systems for Multimodal Question Answering

Related work progresses from task-specific deep multimodal QA models toward LM-based interactive systems with retrieval, planning, collaboration, and tool use. Despite improved cross-modal reasoning and interpretability, both paradigms retain limitations in knowledge coverage, multimodal integration, reasoning stability, and uncertainty modeling.

  • A. Deep Models for Multimodal Question Answering: Deep multimodal QA has advanced interpretable reasoning, knowledge enhancement, and cross-modal fusion through retrieval-augmented generation and multimodal Transformer models.MuRAG integrates image and text retrieval with generation, while another model combines medical diagnoses and chest imaging for multimodal reasoning.
  • A. Deep Models for Multimodal Question Answering: SKURG constructs an entity-centric multimodal evidence graph and aligns it into a unified semantic space to alleviate modality bias in cross-modal retrieval.
  • A. Deep Models for Multimodal Question Answering: Traditional deep QA models depend heavily on task-specific training data, limiting internal knowledge coverage and effectiveness on complex open-world questions.They also face challenges in joint representation learning, cross-modal alignment, and multimodal reasoning.
  • B. Large-Model Interactive Systems for Multimodal Question Answering: The rise of LLMs and VLMs has expanded multimodal applications and motivated interactive systems for increasingly complex problems.
  • B. Large-Model Interactive Systems for Multimodal Question Answering: LM-based multimodal QA has shifted from single-model paradigms toward interactions and controllable reasoning pathways, including prompt–action–tool methods and zero-shot generalization.AI-VQA improves interpretability by exposing intermediate reasoning steps and supporting evidence.
  • B. Large-Model Interactive Systems for Multimodal Question Answering: Interactive systems increasingly combine large language models with retrieval-augmented mechanisms, intelligent routing, and planning to address heterogeneous data and complex multihop queries.RopMura selects knowledge agents and decomposes cross-domain questions into manageable steps for single-hop and multi-hop handling.
  • B. Large-Model Interactive Systems for Multimodal Question Answering: Overall, LM-based MQA has evolved toward planning, collaboration, and tool use, enabling stronger multi-step reasoning and cross-modal evidence integration across diverse domains.However, reliance on LMs can destabilize reasoning, while multimodal fusion, conflict resolution, and explicit uncertainty and error modeling remain challenging.

C. Fuzzy Reasoning and Fuzzy Systems · III. MULTIMODAL GENERATIVE FUZZY SYSTEM (MMGFS) · A. Framework Overview

The paper traces fuzzy systems and rule-based QA as foundations for interpretable reasoning, while identifying the lack of a unified multimodal, rule-driven, and multi-LM framework. MMGFS addresses this gap by combining fuzzy uncertainty modeling with four modules and interactive LM-based processing.

  • C. Fuzzy Reasoning and Fuzzy Systems: Fuzzy sets, including TSK and Mamdani systems, provide the theoretical and representative rule-based foundations for the paper’s fuzzy reasoning framework.Zadeh introduced fuzzy sets in 1965, while the supplementary material details Takagi–Sugeno–Kang and Mamdani fuzzy systems.
  • C. Fuzzy Reasoning and Fuzzy Systems: Traditional rule-based systems offer clear and interpretable QA reasoning but have limited flexibility and adaptability for complex tasks.Fuzzy rules and weighting mechanisms were introduced to improve system adaptability.
  • C. Fuzzy Reasoning and Fuzzy Systems: Rule induction and statistical methods extend rule-based QA toward automatic reasoning-rule discovery, including QArabPro and Rule-KBQA.QArabPro demonstrated rule-based Arabic reading-comprehension QA, while Rule-KBQA combined rule-based reasoning with large language models.
  • C. Fuzzy Reasoning and Fuzzy Systems: Existing approaches lack a unified framework integrating multimodal perception, rule-driven reasoning, and collaborative multi-LM interactions for demanding QA scenarios.The paper identifies consequences for medical diagnosis, cross-domain knowledge integration, and complex decision-making.
  • III. MULTIMODAL GENERATIVE FUZZY SYSTEM (MMGFS): MMGFS models uncertainty, fuzziness, and conflicting information through fuzzy logic to support adaptive, robust, and interpretable multi-LM behavior.Its interactive components collaboratively perceive, reason, and make decisions in dynamic environments.
  • A. Framework Overview: The MMGFS framework comprises the Domain Fuzzification Module, Generative Fuzzy Rule Base, Multi-Domain Multi-Hop Fuzzy Inference Engine, and a fourth core module.The supplied passage names the first three modules and truncates the name of the fourth.
  • A. Framework Overview: Each MMGFS interactive component follows an “LLM/VLM + Prompt” paradigm, using historical context and task objectives to generate prompts and process multimodal external inputs.The generalized operations are 𝒫=𝜓(𝐶,𝐺) and 𝐴=f_LM(I|𝒫), where context accumulates over time while objectives remain constant.

B. Domain Fuzzification Module (DFM)

The Domain Fuzzification Module analyzes multimodal questions through modality-specific processing, structured collaboration, and iterative rumination. It then identifies involved knowledge domains and generates a first-hop question for each domain to support fuzzy reasoning and multi-hop decomposition.

  • Domain Fuzzification Module (DFM): DFM performs domain-oriented fuzzification of multimodal questions through Multi-Modal Rumination and Domain Analysis submodules.Domain uncertainty is addressed through fuzzy reasoning, while logically complex issues are handled through multi-hop decomposition.
  • Multi-Modal Rumination: MIP uses separate modality-specific components, such as Text MIP, Image MIP, and Table MIP, for feature extraction and semantic modeling.This design supports modality-specific optimization and accurate representation of each data type.
  • Multi-Modal Rumination: MMR integrates modality outputs through SIP and MCP, extracting shared information before producing a final cross-modal context.SIP aggregates information consistently supported across modalities, while MCP combines shared information with historical context and the generation goal.
  • Multi-Modal Rumination: The Rumination Mechanism feeds MCP-processed information back to each MIP for iterative refinement of asymmetric multimodal inputs.MCP returns the final processed result after verifying that the input is free from contradictions and semantic errors.
  • Domain Analysis: Domain Analysis contains components that identify question domains from multimodal context and generate a first-hop question for each domain.For K involved domains, it produces the set of first-hop questions x^(1) = [x_1^(1), …, x_K^(1)].

C. Generative Fuzzy Rule Base (GFRB)

The Generative Fuzzy Rule Base (GFRB) extends traditional fuzzy rules with LLM-driven interactive rules for complex multimodal question answering. It combines K domain rules with a Hop Rule to generate next-hop questions and support domain-specific reasoning.

  • Motivation: Traditional fuzzy systems use uncertainty-handling consequents, but their encoded expert knowledge has limited expressiveness for complex multimodal question answering.The proposed approach addresses this limitation with LLM-driven interactive rules.
  • Rule Base Structure: The GFRB contains K domain rules, each targeting problems within a particular domain and providing corresponding solutions.The domain rules are indexed by k = [1, …, K].
  • Rule Base Structure: The GFRB also includes a Hop Rule that generates the next-hop questions needed for iterative reasoning.The next-hop question is denoted x_k^(h+1) for domain k.
  • Rule Semantics: Each domain-rule antecedent uses LLM semantic reasoning to determine membership degrees between the rule conditions and the input.The rule output represents the answer to the current-hop question in domain k.
  • Rule Semantics: The domain-rule objective G_dk governs domain-specific reasoning, while G_h specifies the goal for generating the first-hop question.These objectives distinguish solving domain questions from initiating the hop sequence.

D. Multi-Domain Multi-Hop Fuzzy Inference Engine (MMFIE)

MMFIE is an LLM interactive rule-inference module that performs semantic-level reasoning through complementary domain-rule and hop-rule reasoning. It reduces cross-domain uncertainty and constructs multi-hop reasoning by weighting domain-specific answers and generating further questions when needed.

  • Domain and Hop Rule Reasoning: MMFIE combines domain-rule reasoning, which answers questions across multiple domains, with hop-rule reasoning, which builds a multi-hop reasoning process.Domain reasoning mitigates uncertainty from domain differences, while hop reasoning incrementally uncovers deeper semantic information.
  • Domain Rule Reasoning: Each hop answer is weighted by its domain membership before being passed to the next hop’s domain rules.The weighting is implemented semantically by prepending a phrase that reflects the answer’s degree of domain membership.
  • Domain Rule Reasoning: Domain membership uses six linguistic levels: High, Upper-Medium, Medium, Lower-Medium, Low, and Very-Low.These levels represent the question’s membership degree with respect to each domain.
  • Hop Rule Reasoning: Hop-rule reasoning generates no additional question when the current question is classified as High likelihood of being the last hop; otherwise, it generates a next-hop question.This decision supports continued multi-hop reasoning for membership levels other than High.
  • Inference Procedure: The inference engine first applies multiple domain rules, then uses one shared hop rule to generate questions required for subsequent hops.Domain rules retain the same structure across hops, although their input and output content changes.

E. Domain Rule Fusion Module (DRFM)

The Domain Rule Fusion Module integrates domain-specific answers into a final result using semantic adversarial fusion to improve reliability and robustness. It combines voting, domain-membership filtering, and LLM-based consolidation to select or generate the final answer.

  • E. Domain Rule Fusion Module (DRFM): DRFM fuses answers generated by multiple domain rules into a final result while addressing LLM hallucinations and semantic errors.Each domain rule produces a domain-specific answer after multiple hops before fusion.
  • E. Domain Rule Fusion Module (DRFM): The fusion process first retains the candidate answer receiving the highest votes, then filters low-relevance answers using domain membership values.These two steps are specified in Eqs. (15) and (16).
  • E. Domain Rule Fusion Module (DRFM): A unique highest-vote candidate is directly adopted as the final output, whereas tied candidates undergo membership-degree filtering.The filtering compares the membership degrees of candidates tied for the highest vote count.
  • E. Domain Rule Fusion Module (DRFM): If multiple candidates remain after filtering, LLM interactions fuse them into a consolidated final answer.The resulting output is the final answer after adversarial fusion.

IV. EXPERIMENTS · A. Experimental Setup

The experiments use open-domain and domain-specific multimodal question-answering datasets, with dataset distributions illustrated in Fig. 6. The setup employs multiple LLMs and VLMs to extract semantics across textual, tabular, biological-sequence, visual, and medical-image modalities, with evaluation-metric details provided in the supplementary material.

  • A. Experimental Setup: The experiments cover two dataset categories: open-domain QA with MultimodalQA and WebQA, and domain-specific QA with BioMol-MQA and EHRxQA.The dataset descriptions also reference sample distributions and Table S1.
  • A. Experimental Setup: Fig. 6 shows the question and answer length distributions for the datasets used in the experiments.The figure caption specifies that both questions and answers are represented.
  • A. Experimental Setup: The evaluation includes performance comparisons on the open-domain datasets MultimodalQA and WebQA.These comparisons are presented in Table I.
  • A. Experimental Setup: The evaluation includes performance comparisons on the domain-specific datasets BioMol-MQA and EHRxQA.These comparisons are presented in Table II.
  • A. Experimental Setup: Module ablations are reported for MultimodalQA and WebQA, and separately for EHRxQA and BioMol-MQA.The open-domain results appear in Table III, while the domain-specific results appear in Table IV.
  • A. Experimental Setup: Methods for the evaluation metrics are provided in Part 3 of the supplementary material.The experimental-settings text points to supplementary documentation for these metric methods.
  • A. Experimental Setup: The setup uses GLM 4.5, DeepSeek 3.1, and GPT-4 for semantics from text, tables, and biological sequences.The specified models are glm-4.5flash and deepseek-3.1, alongside GPT-4.
  • A. Experimental Setup: The setup uses glm-4v-flash for visual semantics and VLM-Med for extracting semantics from CT scans.VLM-Med is identified as LLaVA-Med-v1.5-Mistral-7B.

B. Performance Comparison · C. Ablation Study

MMGFS is evaluated across open-domain and domain-specific multimodal QA benchmarks, where it generally surpasses traditional and LM-interactive methods. Ablation results show that shared information, multimodal rumination, and fuzzy reasoning each contribute complementary improvements.

  • B. Performance Comparison: MMGFS is evaluated on MultiModalQA, WebQA, BioMol-MQA, and EHRxQA using answer-accuracy metrics including Acc, Recall, and F1.BioMol-MQA covers drug–target interactions, while EHRxQA covers electronic health records.
  • B. Performance Comparison: Evaluation metrics differ by answer characteristics: short answers favor EM, MRR, and HR@5 for matching, whereas longer answers require fluency-oriented measures.MultiModalQA and BioMol-MQA answers are typically one to two tokens, limiting the usefulness of fluency evaluation.
  • B. Performance Comparison: On open-domain QA, MMGFS outperforms compared methods on most metrics and exceeds traditional and LM-interactive approaches on MultiModalQA accuracy and matching.The comparison is presented in Table I.
  • B. Performance Comparison: MMGFS consistently surpasses LM-interactive methods overall, with multimodal collaboration and fuzzy multi-level uncertainty reasoning improving accuracy, matching quality, and fluency.LM-interactive approaches generally outperform traditional methods.
  • C. Ablation Study: The ablation study progressively adds shared information, multimodal deep mining and rumination, and fuzzy reasoning to a baseline LLM question-answering system.Shared information jointly supplies cross-modal information to the LLM; rumination deepens multimodal processing; fuzzy reasoning adds rule-based inference.
  • C. Ablation Study: 8.29 F1 improvement follows shared-information integration on MultiModalQA, followed by gains of 2.68 from rumination and 3.56 from fuzzy rule-based reasoning.The results show steady improvement as modules are incrementally introduced.
  • C. Ablation Study: 19.72% EM improvement over baseline B follows shared-information integration on BioMol-MQA, with rumination adding 8.74% and rule-based reasoning adding a further gain.Table IV reports these progressively enhanced variants and indicates distinct positive contributions from each module.
  • C. Ablation Study: Together, the three components yield complementary and synergistic effects: shared information supports semantic fusion, rumination deepens understanding and memory, and fuzzy rules strengthen consistency and interpretability.Their combination produces consistent improvements across diverse task settings.

D. Modal Effectiveness Analysis · E. Case Study

Modality ablations show that progressively fusing complementary and structured modalities improves multimodal QA performance and mitigates modality bias. A BioMol-VQA case study further illustrates how MMGFS resolves conflicts through rumination, integrates domain rules with multi-hop inference, and fuses weighted rule outputs into an interpretable answer.

  • D. Modal Effectiveness Analysis: Text+Image improves MultimodalQA F1 by 9.43 over Text alone, showing that image information positively contributes to performance.Further modality additions are evaluated in the same ablation study.
  • D. Modal Effectiveness Analysis: BioMol-MQA HR@5 improves by 9.43 with Graph and by an additional 20.59 after adding Sequence.Graph encodes drug–target interactions, while Sequence represents protein amino acid sequences and drug SMILES.
  • D. Modal Effectiveness Analysis: Across datasets, cross-modal complementarity, semantic alignment, and feature fusion balance information use and suppress reliance on single-modality shortcuts.The reported findings associate fusion with improved performance, cross-modal consistency, robustness, and interpretability.
  • D. Modal Effectiveness Analysis: EHRxQA performance improves by 22.00 when structured Table information is added to image inputs, reducing image noise and supporting more stable reasoning.The table represents patient medical records and provides explicit constraints and logical priors.
  • E. Case Study: In the BioMol-VQA case study, text, sequence, and graph modalities undergo modality-specific extraction, consistency assessment, and three rounds of rumination to resolve semantic conflicts.The resulting cross-modally validated information is passed to later processing stages.
  • E. Case Study: The domain-analysis and fuzzy-rule components structure cross-domain knowledge so it can be integrated and inferred under rule guidance.The MMFIE engine then performs systematic reasoning over domain rules using iterative multi-hop inference.
  • E. Case Study: Rules 1 and 2 reach outputs after two hops, while Rule 3 requires three hops; Rules 2 and 3 converge on “Muscarinic acetylcholine receptor M ”.The example demonstrates hierarchical, interpretable reasoning across multimodal and cross-domain information.
  • E. Case Study: Rule fusion selects “Muscarinic acetylcholine receptor M ” as the final answer because it receives two votes and Rule 2 has the highest activation strength.Semantic voting and activation weighting improve consistency, traceability, and reliability while helping mitigate hallucinations.

F. Further Tests · V. CONCLUSION

Further tests show that MMGFS retains strong reasoning and scalability with controllable computational costs and statistically tested method differences. The conclusion highlights its multimodal fuzzy reasoning advantages, while acknowledging modality and conflict-handling limitations and outlining future directions.

  • F. Further Tests: Supplementary analyses examine the number of fuzzy rules, rumination iterations, and question length as key MMGFS parameters.These tests assess the model’s parameter sensitivity and flexibility.
  • F. Further Tests: Resource, time, and space-complexity evaluations indicate that MMGFS preserves strong reasoning capability with controllable computational and resource costs.The analyses support its scalability and practical applicability.
  • F. Further Tests: Statistical significance was assessed using the Friedman test followed by the Holm post hoc test for multiple comparisons.Detailed results are provided in the supplementary material.
  • V. CONCLUSION: MMGFS is an end-to-end generative fuzzy system for MQA that mitigates modality bias through collaborative multimodal processing.It also uses LM-based fuzzy rules and multi-hop reasoning to address uncertainty and support deeper reasoning.
  • V. CONCLUSION: Experiments on public MQA datasets show superior performance and enhanced robustness compared with existing methods, alongside improved generalization and cognitive consistency.These findings summarize the reported empirical advantages of MMGFS.
  • V. CONCLUSION: MMGFS currently supports a limited range of modalities and remains sensitive to semantic conflicts in multimodal inputs.These limitations may affect reasoning reliability.
  • V. CONCLUSION: Future work targets modality-agnostic representation alignment, conflict-aware reasoning, enhanced fuzzy-rule interpretability, and adaptive continual learning.The goals include supporting new modalities and evolving domain knowledge.
Loading 2608.14584v1…