Source-linked AI summary
Evolution of Multimodal Question Answering: From Modality-Adaptive Extraction to Unified Language Representation
Abdullah Al Shafi
TL;DR
Multimodal QA must reason across heterogeneous sources rather than relying on single-modality inputs. This paper compares MAE, Solar, and UniMMQA, showing an evolution toward unified text-centric reasoning, with UniMMQA providing more robust and scalable benchmark performance despite persistent transformation, alignment, and efficiency challenges.
Problem
Single-source QA systems are limited when relevant information is distributed across heterogeneous modalities such as text, tables, and images.
Method
The paper systematically compares MAE, Solar, and UniMMQA, focusing on modality representation, retrieval, reasoning, and answer extraction.
Results
UniMMQA shows superior and more robust performance across benchmarks, while larger T5 models provide additional performance improvements.
Takeaways & Limitations
The comparison supports a shift from modality-specific processing toward unified text-centric multimodal reasoning with pre-trained language models.
Takeaways & Limitations
Unified multimodal systems still face information loss, error propagation, hallucination, fine-grained information preservation, and computational-efficiency challenges.
Abstract
from arXiv · showhide
The rapid growth of multimodal data has intensified the need for question answering (QA) systems capable of reasoning across heterogeneous sources such as text, tables, and images. In this paper, we present a comprehensive methodological comparison of three influential frameworks, namely Multimodal Adaptive Extraction (MAE), Solar, and UniMMQA, tracing the evolution of multimodal question answering from modality-adaptive pipelines to fully unified architectures. We examine how each approach models cross-modal interactions, transforms heterogeneous inputs, and performs reasoning, highlighting key design differences in modality representation, reasoning, and answer generation. Our analysis demonstrates a clear shift from explicit modality-specific processing toward unified text-centric formulations enabled by pre-trained language models (PLMs). Empirical comparisons across benchmark datasets show that this transition leads to substantial improvements in both Exact Match (EM) and F1-Scores, with UniMMQA achieving the most consistent and scalable performance. Despite these advances, we identify persistent challenges, including information loss during modality transformation, error propagation in multi-stage pipelines, and limitations in capturing fine-grained cross-modal dependencies. Overall, this study provides a deeper understanding of current design trends and offers insights into the future direction of unified multimodal reasoning systems.
I. INTRODUCTION
Multimodal QA addresses the limits of single-source systems by reasoning over text, tables, and images. This review compares three frameworks and traces a shift toward unified, text-centric reasoning with PLMs.
- Single-source QA systems struggle when relevant information is distributed across heterogeneous modalities.
- MMQA and MMCoQA require systems to reason across text, tables, and images, sometimes within multi-turn dialogues.
- The paper systematically compares MAE, Solar, and UniMMQA by examining their handling of heterogeneous modalities in QA.
- The reviewed approaches increasingly reformulate multimodal QA as a text-centric problem using pre-trained language models.
- Existing systems remain limited by information loss during modality transformation, weak cross-modal alignment, and conversational reasoning challenges.
II. LITERATURE REVIEW
The literature spans conversational QA, multimodal reasoning, vision-language pretraining, MLLMs, and grounded structured reasoning. Across these areas, researchers seek stronger cross-modal understanding while facing grounding, supervision, and transfer limitations.
- Conversational QA benchmarks established the need to model dialogue context, while many systems still focus only on text.
- MultimodalQA and ManyModalQA require compositional reasoning across textual, tabular, and visual information.
- Vision-language models such as BLIP-2 and PaLI combine pretrained vision encoders with language models for zero-shot and few-shot tasks.
- MLLMs process multiple input types and reasoning tasks but can produce hallucinated outputs in complex multimodal contexts.
- Grounded structured reasoning methods can improve performance but often require extra supervision and specialized designs.
III. METHODOLOGICAL EVOLUTION
Methodological development progresses from modality-adaptive extraction toward unified language-based reasoning. MAE retrieves multimodal evidence but selects one modality for extraction, creating limitations for joint reasoning.
- Recent multimodal QA approaches evolve from MAE’s modality-adaptive architecture toward unified language frameworks and end-to-end reasoning models.
- MAE: MAE decomposes conversational multimodal QA into question understanding, evidence retrieval, and adaptive answer extraction.
- MAE: MAE transforms text, linearized tables, and image features into representations used to retrieve relevant multimodal evidence.
- MAE: MAE predicts the most suitable modality, selects its extractor, and scores candidate answers using retrieval relevance, modality confidence, and extraction probability.
- MAE: Because extraction relies on one predicted modality, MAE cannot fully support questions requiring complementary information from multiple modalities.
B. Unified
Solar converts heterogeneous inputs into a unified textual space and applies retrieval-augmented text generation. This improves modality-agnostic processing but remains vulnerable to lossy image conversion and accumulated retrieval errors.
- Solar: Solar transforms text, tables, and images into a unified linguistic representation for a text-only QA pipeline.
- Solar: Solar linearizes tables into natural-language sentences using deterministic templates.
- Solar: Solar uses global captions and local image representations because image-to-text conversion lacks a native linguistic structure and is inherently lossy.
- Solar: Solar incorporates prior dialogue turns into the query representation to capture coreference, ellipsis, and conversational dependencies.
- Solar: Solar reframes multimodal QA as retrieval-augmented text generation through sequential retrieve, rank, and generate stages.
- Solar: The multi-stage pipeline accumulates errors because it cannot retrieve evidence missing during retrieval.
C. Unified Language Representation with Explicit Rationale-enhanced Reasoning: UniMMQA [8]
UniMMQA converts text, tables, and images into a unified textual representation, then uses rationale-enhanced cross-modal reasoning and T5 for answer generation. Its transformations improve coverage of visual and structural information but introduce length, efficiency, and hallucination risks.
- Unified representation: UniMMQA transforms heterogeneous inputs—text, tables, and images—into unified text before processing them with a generative PLM.The transformed components are concatenated into one sequence for T5-based answer generation.
- Modality transformation: Image-to-text transformation combines OCR with diversified BLIP captions generated using Top-K and Top-p sampling.Multiple captions capture objects and background details that a single caption may omit, while OCR preserves visible textual content.
- Modality transformation: Position-enhanced table linearization preserves table content and structural elements through row-based serialization and explicit indicators.The method avoids template-based summarization because it may discard task-relevant attributes.
- Rationale-enhanced reasoning: A multimodal rationale generator models relationships between visual features and textual representations before answer generation.CLIP extracts visual features, while a language encoder represents passages and linearized tables for rationale production.
- Answer generation: The final stage concatenates the question, transformed modalities, and rationale into one sequence and fine-tunes T5 with negative log-likelihood.The decoder produces output tokens matching the ground-truth answer sequence.
- Limitations: UniMMQA may miss fine-grained visual details, increase input complexity through table linearization, and introduce hallucinated information through captioning or rationale generation.These issues can misguide reasoning and reduce efficiency or answer quality.
A. Datasets
The evaluation uses three public multimodal QA benchmarks covering image, text, table, and conversational settings. Their sizes and cross-modal composition vary substantially.
- Datasets: Experiments use ManymodalQA, MultimodalQA, and MMConvQA as public benchmark datasets.The benchmarks cover multimodal and conversational question answering.
- Datasets: ManymodalQA contains 10,190 questions spanning 2,873 images, 3,789 text instances, and 3,528 tables.The questions are split into training and development sets.
- Datasets: MultimodalQA contains 29,918 question-answer pairs across image, table, and text modalities, with 35.7% requiring cross-modal reasoning.
- Datasets: MMConvQA extends MultimodalQA with 1,179 conversations and 5,753 questions.
B. Evaluation Metrics
The study evaluates multimodal QA systems with Exact Match and F1, while comparing them against several retrieval, decomposition, sequence-to-sequence, graph, and multimodal baselines.
- Evaluation Metrics: Exact Match assigns 1 only when the normalized prediction exactly matches the ground-truth answer, and 0 otherwise.
- Evaluation Metrics: F1 measures token-level overlap using precision and recall, providing partial credit for non-exact answers.
- Baseline Models: Compared systems include ORConvQA, ManyModalQA, Implicit-Decomp, MMQA-T5-Large, PReasM, SKURG, and MGT.
- Baseline Models: ORConvQA combines a trainable retriever, reranker, and reader, whereas ManyModalQA routes questions to modality-specific QA systems after classification.
- Baseline Models: Implicit-Decomp generates programs that execute single-modal and multimodal QA components across multiple reasoning steps.
- Baseline Models: PReasM enhances T5 with synthetic tabular data, while SKURG and MGT use graph-based structures to model multimodal connections and interactions.
- Results: Table I reports experimental results on the MultimodalQA dataset.
V. EXPERIMENTAL RESULT
Across MultimodalQA, MMCoQA, and ManymodalQA, results show progression from weak and modular baselines toward unified systems, with UniMMQA generally delivering the strongest and most scalable performance.
- UniMMQA and Solar outperform weaker multimodal baselines on MultimodalQA, while structural approaches remain limited without effective joint representation learning.The comparison emphasizes the value of unified textual and tabular representations over separate components or structural connections alone.
- MAE substantially improves EM and F1 over MMCoQA baselines through conversational comprehension and adaptive extraction, but unified systems perform better.Its modular pipeline provides a strong intermediate step between weak baselines and unified reasoning systems.
- Solar further improves performance on development and test sets, while UniMMQA remains superior across testing conditions.The results associate stronger multimodal integration and retrieval-augmented reasoning with improved answer quality, without matching UniMMQA.
- UniMMQA improves as the underlying T5 model grows from T5-Base to T5-Large and T5-3B, indicating scalable use of larger pre-trained language models.The smallest UniMMQA model already exceeds Solar, while larger variants provide additional gains.
- On ManymodalQA, UniMMQA is competitive with or generally superior to the baseline without explicit question-type classification, while larger models improve EM and F1.ManymodalQA with type remains a strong heuristic baseline in specific situations, showing that explicit modality supervision can still help.
B. Modality-wise Performance
UniMMQA’s modality-wise performance varies with dataset structure and modality complexity: textual inputs are often strongest, while transformed images, tables, and conversational data remain challenging.
- UniMMQA performs better on text-based than image- and table-based MultimodalQA questions, reflecting restrictions in visual and tabular processing.Converting images and tables into text can remove information needed for accurate reasoning.
- ManymodalQA table questions outperform its image- and text-based questions because the dataset uses relatively simple tables with few rows.Longer and more complex passages create greater context and reasoning difficulty.
- MMConvQA table-based questions show a major performance drop, whereas image- and text-based questions perform better.Conversational tabular reasoning must combine structured data with evolving dialogue information.
- Overall, UniMMQA performance depends on modality structure and dataset-specific success in linearizing and combining heterogeneous inputs.The results motivate stronger preservation of structural and visual semantics and improved multimodal fusion.
VI. CONCLUSION
The paper concludes that multimodal QA has evolved from modality-specific pipelines toward unified text-centric frameworks. UniMMQA offers more robust and scalable performance, but information preservation, hallucination, and efficiency remain open challenges.
- The comparison traces multimodal QA’s evolution from MAE’s modular pipeline to Solar’s unified textual space and UniMMQA’s rationale-enhanced framework.The progression concerns how heterogeneous modalities are represented, retrieved, and reasoned over.
- UniMMQA achieves more robust and scalable benchmark performance through richer modality-to-text transformations and explicit rationale generation.The conclusion identifies unified PLM-based approaches as more effective for multimodal reasoning.
- Open challenges include preserving fine-grained multimodal information, mitigating hallucination, and improving computational efficiency.These limitations define directions for future unified multimodal reasoning systems.