PaperlayerView exact revision on arXiv

Source-linked AI summary

HAM-RAG: Hierarchy-Aware Multimodal RAG for Structure-Faithful Interleaved Generation

Yin Li, Ziyang Hu, Zhiyu Guo, Xiangyu Liu, Wenbin Li, Boo-Ho Yang, Rav Lawana, Ziyue Li, Wei Zeng, Fugee Tsung

arXiv:2608.14032v1cs.IRcs.AI

TL;DR

Flat multimodal RAG can lose document organization and local text-image relations needed for faithful evidence selection and placement. HAM-RAG makes hierarchy a grounding signal across retrieval and generation, improving the main multimodal average by 17.3% over the strongest non-hierarchical baseline across matched backbones.

  • Problem

    Multimodal RAG must preserve textual scope, image grounding, and image placement within structured documents, not merely retrieve globally relevant text and images.

  • Method

    HAM-RAG enriches textual and visual evidence with document hierarchy, source position, and local cross-modal context throughout retrieval and generation.

  • Results

    17.3%: HAM-RAG improves the main multimodal average over the strongest non-hierarchical baseline across matched backbones.

  • Takeaways & Limitations

    Document hierarchy is a key grounding signal for faithful image selection, placement, local text-image alignment, and coherent multimodal generation.

Abstract

from arXiv · show

Existing multimodal RAG methods often flatten structured documents into isolated text and image units, weakening the source organization and local text-image logic needed for faithful evidence selection and placement. We propose HAM-RAG, a Hierarchy-Aware Multimodal RAG framework for structure-faithful interleaved generation. HAM-RAG uses document hierarchy as a grounding signal across retrieval and generation, contextualizing textual and visual evidence and preserving source position and local text-image relations in the prompt. We further introduce HAM-Bench, covering Wukong, Wiki, arXiv, and Recipe across game walkthroughs, web pages, scientific papers, and step-wise recipe documents. Across multiple backbones, HAM-RAG improves the main multimodal average by 17.3% over the strongest non-hierarchical baseline. On Wukong, HAM-RAG improves Img-CBS by 24.2% over the strongest non-hierarchical baseline, demonstrating substantially better local text-image alignment. The main experiments and ablation study together demonstrate that document hierarchy is a key grounding signal for faithful image selection, placement, and local text-image alignment. These findings highlight the value of hierarchy-aware grounding for reliable multimodal assistants that generate answers faithful to the source organization, procedural structure, and local text-image evidence of structured documents, such as technical manuals, maintenance guides, and industrial SOPs. The code is available at https://github.com/MCCodeAI/HAM-RAG.git.

1. INTRODUCTION

HAM-RAG treats document hierarchy as a grounding signal for structure-faithful multimodal RAG, preserving textual scope, image grounding, placement, and local text-image logic. HAM-Bench evaluates these capabilities across diverse structured documents, where HAM-RAG yields substantial gains in multimodal quality and visual alignment.

  • Motivation: Structured-document multimodal RAG must preserve contextual organization and local text-image logic, not merely retrieve topically relevant text and images.Images can be wrong when drawn from another section or step, and correct images can mislead when placed beside unrelated text.
  • HAM-RAG: HAM-RAG contextualizes retrievable text and images with document hierarchy, including headings, sections, document position, nearby text, captions, and visual descriptions.This representation supports textual scope, image grounding, and image placement in generated answers.
  • HAM-Bench: HAM-Bench covers web pages, scientific papers, step-wise recipes, and game walkthroughs, with local text-image alignment metadata for structure-preserving generation evaluation.The benchmark includes both natively structured documents and documents whose structure is reconstructed from source materials.
  • Results: 17.3%: HAM-RAG improves the main multimodal average across matched backbones over the strongest non-hierarchical baseline.On Wukong, Img-CBS increases by 24.2% over the strongest non-hierarchical baseline, demonstrating stronger visual grounding and local text-image alignment.

2. RELATED WORK

Prior multimodal RAG and document-understanding methods retrieve multimodal evidence or model document structure, but generally do not preserve source organization and local text-image logic during interleaved generation. HAM-RAG makes document hierarchy a core grounding signal for textual and visual evidence, addressing this gap alongside benchmarks that do not explicitly measure section-hierarchy preservation.

  • Multimodal RAG: Multimodal RAG extends text-only retrieval by retrieving textual and visual evidence for answer generation, with later methods supporting multimodal answers and preserving page appearance.MuRAG supports text answer generation, while MuRAR, M2RAG, and VisRAG extend multimodal retrieval-augmented generation toward multimodal answers or page-preserving retrieval.
  • Structure-aware retrieval: Structure-aware methods address flat chunking through hierarchical retrieval, page-level visual preservation, or query-centric graphs for multimodal cross-page evidence.Wiki-LLaVA, M3DocRAG, and MLDocRAG represent distinct approaches to organizing or preserving multimodal document evidence.
  • HAM-RAG: Existing structure-aware approaches primarily target document QA, long-document retrieval, or multimodal understanding rather than structure-faithful interleaved generation.HAM-RAG instead treats document hierarchy as a core grounding signal for both textual and visual evidence, rather than auxiliary metadata.
  • Benchmarks: Multimodal QA and retrieval benchmarks evaluate multimodal evidence use, evidence selection, and interleaved text-image answers, but do not explicitly measure preservation of section hierarchy.WebQA and MMCoQA focus on multimodal-evidence question answering, while MRAMG-Bench and MMDocRAG extend evaluation toward retrieval, evidence selection, and interleaved generation.

3. HAM-RAG FRAMEWORK

HAM-RAG performs multimodal retrieval-augmented generation over hierarchical documents, using document structure as a shared signal for evidence representation, retrieval, and prompt construction. Its offline indexing and online retrieval-generation stages preserve source position, hierarchy, and local text-image relations for structure-faithful generation.

  • Overview: HAM-RAG combines offline hierarchy-aware indexing with online multimodal retrieval and generation, unlike flat RAG pipelines that isolate passages and images.Document hierarchy serves as a shared signal across evidence representation, retrieval, and prompt construction.
  • Hierarchy Construction: Each document becomes a hierarchy tree whose internal nodes represent documents, sections, subsections, or steps, while leaves represent retrievable text and image units.Edges encode parent–child relations, and hierarchy is recovered from markup, headings, numbering, figure-caption links, layout, and source order.
  • Evidence Representation: Each leaf receives an evidence object containing content, modality, hierarchy path, ancestor context, nearby text-image context, and local relation.Local relations can identify illustrative images, figure-caption membership, or support for a preceding paragraph, making each unit’s semantic scope explicit.
  • Offline Indexing: Offline indexing serializes text with title, heading path, section context, and neighboring evidence, while images retain position, captions, and context-aware visual descriptions.Hierarchy therefore influences retrieval through the indexed representation rather than being appended only after retrieval.
  • Online Retrieval and Generation: Online retrieval selects top-ranked text and image evidence under separate budgets k_t and k_v, retaining relevance, source identity, position, hierarchy, and local relations.The structured prompt exposes headings, neighboring context, image references, captions, positions, and descriptions so generation can align images with statements and placement.

4. HAM-BENCH

HAM-Bench evaluates whether multimodal RAG preserves document hierarchy, local context, and text-image associations during retrieval and interleaved generation. Its four subsets span scientific, procedural, web, and game-walkthrough documents with complementary structural challenges.

  • Evaluation framework: Table 1 distinguishes multimodal evidence, interleaved generation, structure use, hierarchy reconstruction, and hierarchy-aware evaluation using Yes, Partial, or No support labels.Yes indicates full support, Partial limited or indirect support, and No absence.
  • Benchmark purpose: HAM-Bench tests hierarchy-aware retrieval and interleaved text-image generation while preserving contextual organization and local text-image logic.It is built on hierarchy-preserving multimodal corpora and targets capabilities beyond evidence usage or multimodal answer quality.
  • Benchmark design: Unlike prior benchmarks, HAM-Bench preserves document hierarchy, reconstructs flattened sources, and evaluates structure-aware retrieval and generation.Reconstruction restores document boundaries, section structures, and local text-image alignments.
  • Dataset composition: The benchmark covers Wukong walkthroughs, WikiWeb2M pages, section-structured arXiv PDFs, and step-wise hierarchical Recipe documents.These subsets represent game walkthroughs, web pages, scientific papers, and procedural recipe documents.
  • Structural challenges: The subsets present complementary challenges: deep hierarchies and dense visuals in arXiv, procedural order and local alignment in Recipe and Wukong, and shallower structure in Wiki.Wukong additionally requires multi-document reasoning, while Wiki contains fewer images.

5. EXPERIMENTS

Experiments compare hierarchy-aware and non-hierarchical paradigms under matched retrieval and generation settings, evaluating retrieval, multimodal fidelity, ablations, and operational robustness. HAM-RAG consistently improves multimodal performance and local image-text alignment across backbones, while maintaining low failure rates and moderate cost.

  • Experimental Setup: All methods share BGE-M3 retrieval embeddings, prompts, scripts, budgets, and failed-case handling, isolating evidence representation as the main comparison variable.Baselines use flat evidence representations, whereas HAM-RAG adds hierarchy-aware context to text and image units.
  • Evaluation: Evaluation measures text and image retrieval coverage alongside Image F1, Img-CBS, Overall Quality, and the four-dataset, three-metric MM Avg.TCR@15 measures top-15 text evidence sufficiency, IR measures retained reference-image coverage, and Img-CBS evaluates image placement in local context.
  • Main Results: 17.3% relative improvement raises MM Avg. from 54.68 to 64.12 across nine matched backbones, with HAM-RAG outperforming its LLM-Based counterpart in every comparison.The result is consistent across proprietary and open-source generator families and different model scales.
  • Main Results: 24.2% improvement raises Wukong’s best Img-CBS from 50.82 to 63.10 over the strongest non-hierarchical baseline, demonstrating better local image-text alignment.HAM-RAG achieves higher TCR@15 on all four subsets and higher IR on three, with arXiv as the exception for IR.
  • Ablation Study: A GPT-4o ablation on 540 Wukong questions finds the full model highest on all three metrics, while weakening image and text hierarchy reduces performance.Replacing SI with LI sharply reduces both image metrics, and LI+ST produces the lowest Average Score, showing complementary image- and text-side contributions.
  • Robustness and Cost: 0.067% failure rate covers 38 failures in 57,123 runs; HAM-RAG has 2 failures, fewer than LLM-Based’s 29 and MLLM-Based’s 7, at 3.578 cents per question.Both HAM-RAG failures occur on arXiv, and its cost is lower than LLM-Based and MLLM-Based but higher than Rule-Based.

6. CONCLUSION

HAM-RAG targets faithful interleaved generation for structured multimodal documents by preserving document organization, local text-image relations, and multi-level context across retrieval and generation. HAM-Bench provides a unified benchmark for evaluating this setting across diverse structured-document scenarios.

  • Motivation: HAM-RAG addresses flat retrieval’s weakening of dependencies among sections, local contexts, and visual evidence.The framework is designed for faithful interleaved generation over structured multimodal documents.
  • HAM-RAG: HAM-RAG incorporates multi-level document context into text and image representations.This mechanism preserves source organization and local text-image relations across retrieval and generation.
  • HAM-Bench: HAM-Bench provides a unified benchmark across diverse structured-document scenarios.
Loading 2608.14032v1…