PaperlayerView exact revision on arXiv

Source-linked AI summary

MuRAR: A Simple and Effective Multimodal Retrieval and Answer Refinement Framework for Multimodal Question Answering

Zhengyuan Zhu, Daniel Lee, Hong Zhang, Sai Sree Harsha, Loic Feujio, Akash Maharaj, Yunyao Li

arXiv:2408.08521v2cs.IRcs.CL

TL;DR

Existing QA systems often focus on text and do not adequately address comprehensive multimodal answers, particularly for step-by-step tutorials. MuRAR retrieves multimodal data to refine text answers, and human evaluation found its multimodal answers more useful and readable than plain text answers.

  • Problem

    Existing approaches focus largely on text answers and do not adequately address complex questions requiring step-by-step explanations that integrate multimodal content.

    Source: §1 · p. 1

  • Method

    MuRAR generates an initial text answer, retrieves multimodal data relevant to its snippets, and refines the answer by integrating that data.

    Source: §1 · p. 1

  • Results

    Human evaluation found MuRAR’s multimodal answers more useful, readable, and relevant than text-only responses, and consistently preferred over plain text answers.

    Source: §1 · p. 2, §6 · p. 7

  • Takeaways & Limitations

    Other enterprise assistants can adapt MuRAR by collecting topic-specific multimodal data and fine-tuning a topic-specific text retrieval model.

    Source: §1 · p. 2

    Source attribution can limit recall when relevant multimodal data lies outside the same section or web document, and duplicated content can reduce readability and clarity.

    Source: §5.4 · p. 7

Abstract

from arXiv · show

Recent advancements in retrieval-augmented generation (RAG) have demonstrated impressive performance in the question-answering (QA) task. However, most previous works predominantly focus on text-based answers. While some studies address multimodal data, they still fall short in generating comprehensive multimodal answers, particularly for explaining concepts or providing step-by-step tutorials on how to accomplish specific goals. This capability is especially valuable for applications such as enterprise chatbots and settings such as customer service and educational systems, where the answers are sourced from multimodal data. In this paper, we introduce a simple and effective framework named MuRAR (Multimodal Retrieval and Answer Refinement). MuRAR enhances text-based answers by retrieving relevant multimodal data and refining the responses to create coherent multimodal answers. This framework can be easily extended to support multimodal answers in enterprise chatbots with minimal modifications. Human evaluation results indicate that multimodal answers generated by MuRAR are more useful and readable compared to plain text answers.

1 Introduction

Existing QA systems often produce text-only or appended-media responses, leaving complex, step-by-step answers that integrate multiple modalities insufficiently addressed. MuRAR retrieves relevant multimodal data to refine text answers, and human evaluation found its multimodal answers consistently preferred over plain text.

1 Introduction

  • Earlier systems mainly generated plain-text answers or appended a retrieved image or video, leaving complex, multi-step answers integrating multiple modalities inadequately addressed.Such multimodal information can be crucial for understanding complex, domain-specific topics in enterprise documentation.Source: §1 · p. 1
  • MuRAR addresses two challenges: retrieving multimodal data relevant to a question and integrating that data into a coherent answer.The framework has text-answer generation, source-based multimodal retrieval, and multimodal answer refinement components.Source: §1 · p. 1
  • Human evaluation on 300 annotated question-answer pairs found that MuRAR retrieved useful, relevant multimodal data while maintaining readability, and its multimodal answers were consistently preferred over plain text.The framework was applied to an enterprise-level AI assistant.Source: §1 · p. 2
  • Other enterprise assistants can adapt MuRAR by collecting topic-specific multimodal data and fine-tuning a topic-specific text retrieval model.Source: §1 · p. 2

2 Design of MuRAR

MuRAR first generates a text answer, retrieves multimodal data linked to its snippets, and refines the answer by integrating that data. The design uses source attribution and section-level retrieval to narrow multimodal search, then guides an LLM to produce a coherent response.

2 Design of MuRAR

  • MuRAR’s design organizes retrieved text snippets, images, tables, and videos relevant to a user question into a coherent, informative multimodal answer.Its pipeline generates an initial text answer, retrieves multimodal data relevant to its snippets, then refines the answer with that data.Source: §2 · p. 2, Figure 1 · p. 3
  • Directly prompting an LLM to handle text and multimodal data is described as ineffective because the task requires choosing what to reference, display, and where to place it.The authors say this complexity can result in low-quality multimodal answers.Source: §2 · p. 2

2.1 Text Answer Generation

  • Text answer generation retrieves relevant document snippets with a fine-tuned embedding model and FAISS, then prompts an LLM with the top five snippets to draft an answer.The embedding model is fine-tuned on annotated query–snippet relevance labels; retrieval uses cosine similarity.Source: §2.1 · p. 2

2.2 Source-Based Multimodal Retrieval

  • Source-based retrieval links each answer snippet to its closest source, then searches multimodal data in that source’s section and ranks it against the snippet.A source is assigned only when the highest cosine similarity is at least 0.6. Retrieval uses contextual text and generated captions or summaries; duplicate selections are reduced to the highest-similarity result.Source: §2.2 · pp. 2–3

2.3 Multimodal Answer Refinement

  • Answer refinement prompts an LLM with the question, initial answer, and retrieved media with contextual features, using placeholders to integrate relevant content into a readable response.The LLM is instructed to replace placeholders with descriptions and revise the associated text; media URLs become interactive HTML elements for display.Source: §2.3 · p. 3

3 User Interface

The prototype integrates MuRAR into an AI assistant in Adobe Experience Platform. It retrieves text and relevant multimodal content for a query, then presents that content as interactive elements.

3 User Interface

  • The prototype places MuRAR in an AI assistant within Adobe Experience Platform, answering a schema-creation tutorial query using Adobe Experience League documents.Source: §3 · p. 3
  • For the example query, the assistant retrieves text snippets, attributes answer snippets to sources, and selects a schema-workspace screenshot and video tutorial.The retrieved multimodal elements appear in the refined answer; clicking one opens the content in a full-size pop-up.Source: §3 · pp. 3–4
  • The demonstrated interface combines multimodal data retrieval with answer refinement to present multimodal answers.Source: Figure 2 · p. 4

4 Data Collection

The authors collected a multimodal document corpus and a question-answer dataset to develop MuRAR and conduct human evaluation. The question-answer evaluation compares text-based and multimodal answers generated with GPT-3.5 and GPT-4.

4 Data Collection

  • The study assembled a multimodal document dataset and an annotated question-answer dataset for framework development and human evaluation.Source: §4 · p. 4

4.1 Multimodal Document Dataset

  • The document corpus covers 2,173 web documents and four modalities: text, image, table, and video.It includes 18,071 text snippets, 6,320 image entries, 2,646 table entries, and 253 video entries.Source: §4.1 · pp. 4–5, Table 1 · p. 5

4.2 Multimodal Question Answering Dataset

  • Of 764 real customer questions answered with MuRAR, 306 received multimodal answers; 150 were sampled for evaluation with GPT-3.5 and GPT-4.Each model generated 150 text-based and 150 multimodal answers, yielding 300 question-answer pairs.Source: §4.2 · p. 5

5 Human Evaluation

The human evaluation measured the added value of integrating multimodal data rather than the quality of the text itself. Ratings and preferences favored MuRAR’s multimodal answers, while the evaluation also identified retrieval and readability limitations.

5.1 Study Setup

  • The study compared GPT-3.5 and GPT-4 outputs, generating text-only and multimodal answers for each question; eight expert annotators evaluated each question-answer pair, with at least three experts per pair.The annotators held advanced degrees in computer science and had substantial experience in natural language processing.Source: §5.1 · p. 5

5.2 Evaluation Schema

  • Annotators rated usefulness, readability, and relevance on 5-point Likert scales, then chose text-only, multimodal, or equally effective responses.The study assessed the added value of multimodal integration, not the quality of the text content itself.Source: §5.2 · pp. 5–6

5.3 Results

  • Usefulness averaged 3.47, readability 3.63, relevance 3.78, and preference rating 0.86; GPT-4 increased all metrics over GPT-3.5.The evaluation results are reported for answers generated by both models.Source: Table 2 · p. 6, §5.3 · p. 6
  • Overall Krippendorff’s alpha was 0.4179, while top-two Cohen’s kappa was 0.71; annotators did not undergo formal training.The authors characterize overall agreement as moderate and top-two agreement as substantial.Source: Table 3 · p. 6, §5.3 · p. 6
  • Score standard deviations ranged from 0.8519 to 0.9496; Annotators 2 and 4 averaged 3.0619 (SD = 0.5648) and 3.3836 (SD = 0.253).The authors suggest differences in scoring patterns among a subset of annotators may account for the lower overall alpha.Source: Table 4 · p. 6, §5.3 · p. 6

5.4 Limitations

  • Source attribution can miss relevant media outside the source section or document, while duplicated multimodal content can reduce answer readability and clarity.The authors identify these as issues in retrieval and answer refinement, respectively.Source: §5.4 · p. 7

6 Conclusion and Future Work

MuRAR adds images, tables, and videos to text-based answers, and human evaluations found its multimodal answers more useful, readable, and relevant than text-only responses. Future work targets answer quality and user experience.

6 Conclusion and Future Work

  • Human evaluations found MuRAR’s multimodal answers more useful, readable, and relevant than text-only responses.The framework incorporates images, tables, and videos into text-based responses.Source: §6 · p. 7
  • Planned quality improvements include broadening the multimodal document dataset, replacing reliance on proprietary models, and addressing relatively low multimodal retrieval recall.Source: §6 · p. 7
  • Planned user-experience improvements include letting videos jump to question-relevant segments and making interaction more intuitive and seamless.Source: §6 · p. 7

7 Ethical Considerations

The paper identifies content appropriateness and privacy as risks when MuRAR is applied to sources beyond the enterprise documents used in this work. It calls for content safeguards and privacy-preserving techniques in those broader settings.

7 Ethical Considerations

  • The Adobe Experience Platform repository used here is considered free from harmful content, but poorly regulated or curated sources may yield inappropriate, offensive, or irrelevant material.Source: §7 · p. 7
  • The paper says broader deployments need robust content filtering, validation, and contextualization, alongside privacy-preserving retrieval and integration.Source: §7 · p. 7
  • The framework’s flexibility across data sources creates a risk of exposing sensitive information, although the enterprise documents used here contained no sensitive content.Source: §7 · p. 7

A.1 Multimodal Scraper Design

The scraper collects modality-specific content and surrounding context, together with document and section metadata, for images, videos, and tables.

A.1 Multimodal Scraper Design

  • For images, the scraper collects the image link, text between neighboring images, document title, section header, and document URL.Source: §A.1 · p. 9
  • For videos, it collects the video URL, preceding context text, transcript, document title, section headers, and document URL.Source: §A.1 · p. 9
  • For tables, it collects table content as a JSON string, preceding context text, document URL, title, and section header.Source: §A.1 · p. 9

A.2 Human Evaluation Metrics

The human-evaluation rubric defines usefulness, readability, and relevance for multimodal elements, and records annotators’ overall preference between text-only and multimodal answers.

A.2 Human Evaluation Metrics

  • Usefulness measures multimodal elements’ contribution to comprehension, with scores from 1 (not at all useful) to 5 (extremely useful).Source: §A.2 · p. 9
  • Readability assesses integration with the text, ranging from 1 (severely impairs reading flow) to 5 (significantly enhances the reading experience).Source: §A.2 · pp. 9–10
  • Table 5 presents per-annotator average scores and preferences.Source: Table 5 · p. 10
  • Relevance measures how closely multimodal elements relate to the text, on a scale from 1 (completely irrelevant) to 5 (perfectly relevant).Source: §A.2 · p. 10
  • Annotators record whether they prefer the text-only answer, the multimodal answer, or find them equally effective.Source: §A.2 · p. 10

A.3 Additional Human Evaluation Results

The appendix reports per-annotator average scores and preferences, and presents simplified versions of the text-generation and multimodal-refinement prompts.

A.3 Additional Human Evaluation Results

  • Table 5 presents the average score and preference for each annotator.Source: §A.3 · p. 10

A.4 Prompts

Loading 2408.08521v2…