RAG-Anything: All-in-One RAG Framework

Zirui Guo; Xubin Ren; Lingrui Xu; Jiahao Zhang; Chao Huang

Source-linked AI summary

RAG-Anything: All-in-One RAG Framework

Zirui Guo, Xubin Ren, Lingrui Xu, Jiahao Zhang, Chao Huang

arXiv:2510.12323v1Oct 14, 2025cs.AI

TL;DR

Existing RAG frameworks remain largely text-only despite real-world documents combining textual, visual, tabular, and mathematical information. RAG-Anything unifies these modalities through dual graphs and hybrid retrieval, achieving superior benchmark performance with particularly pronounced gains on long documents. Its analysis also identifies text-centric retrieval bias and rigid spatial processing as continuing challenges for multimodal RAG systems.

Problem
Existing RAG frameworks focus on text-only knowledge, while real-world documents combine textual, visual, structured, and mathematical information that requires multimodal retrieval.
Method
RAG-Anything represents multimodal content as interconnected knowledge entities using dual graphs and combines structural navigation with semantic matching for cross-modal retrieval.
Results
RAG-Anything achieves superior performance across DocBench and MMLongBench, with gains becoming particularly pronounced on long-context materials.
Takeaways & Limitations
The framework provides unified multimodal knowledge access by preserving structural relationships across text, images, tables, and equations.
Takeaways & Limitations
Failure analysis identifies text-centric retrieval bias and rigid spatial processing on non-standard layouts and structurally ambiguous tables.

Abstract

from arXiv · show

Retrieval-Augmented Generation (RAG) has emerged as a fundamental paradigm for expanding Large Language Models beyond their static training limitations. However, a critical misalignment exists between current RAG capabilities and real-world information environments. Modern knowledge repositories are inherently multimodal, containing rich combinations of textual content, visual elements, structured tables, and mathematical expressions. Yet existing RAG frameworks are limited to textual content, creating fundamental gaps when processing multimodal documents. We present RAG-Anything, a unified framework that enables comprehensive knowledge retrieval across all modalities. Our approach reconceptualizes multimodal content as interconnected knowledge entities rather than isolated data types. The framework introduces dual-graph construction to capture both cross-modal relationships and textual semantics within a unified representation. We develop cross-modal hybrid retrieval that combines structural knowledge navigation with semantic matching. This enables effective reasoning over heterogeneous content where relevant evidence spans multiple modalities. RAG-Anything demonstrates superior performance on challenging multimodal benchmarks, achieving significant improvements over state-of-the-art methods. Performance gains become particularly pronounced on long documents where traditional approaches fail. Our framework establishes a new paradigm for multimodal knowledge access, eliminating the architectural fragmentation that constrains current systems. Our framework is open-sourced at: https://github.com/HKUDS/RAG-Anything.

1 INTRODUCTION

RAG-Anything addresses the mismatch between text-only retrieval and multimodal documents by unifying heterogeneous content and enabling structure-aware, cross-modal retrieval. Its experiments report superior performance, with stronger gains on long-context materials, while ablations identify graph-based representation as the primary source of improvement.

Introduction: Existing text-only RAG systems discard non-textual information or flatten it into inadequate textual approximations, despite documents combining text, visuals, tables, and equations.This limitation is especially consequential in academic, financial, and technical domains where essential insights may be encoded in non-textual formats.
Technical Challenges: Multimodal RAG requires unified representation, structure-aware decomposition, and cross-modal retrieval because relevant evidence spans heterogeneous modalities and sections.These challenges require preserving visual semantics, spatial and hierarchical relationships, and coordinated reasoning across information sources.
Our Contributions: RAG-Anything constructs complementary knowledge graphs that preserve cross-modal relationships and fine-grained textual semantics across visual, structured, and mathematical content.The dual-graph design maintains semantic integrity across modalities while supporting cross-modal reasoning.
Our Contributions: RAG-Anything combines structural knowledge navigation with semantic similarity matching to retrieve relevant content lacking direct graph connections.Modality-aware query processing and cross-modal alignment allow textual queries to access visual and structured information.
Experimental Validation: RAG-Anything achieves superior performance on DocBench and MMLongBench, with advantages becoming more pronounced as content length increases.The evaluation targets challenging multimodal benchmarks and long-context materials.

2 THE RAG-ANYTHING FRAMEWORK

RAG-Anything provides a unified retrieval pipeline for heterogeneous multimodal documents by combining modality-aware knowledge representation, dual-graph construction, and hybrid retrieval. It preserves cross-modal context and textual semantics while supporting structural and similarity-based search.

2.1 Motivating RAG-Anything: RAG-Anything unifies multimodal knowledge retrieval through universal indexing, cross-modal adaptive retrieval, and knowledge-enhanced response generation.The framework is designed for heterogeneous data modalities and document formats within one retrieval pipeline.
2.2 Universal Representation for Heterogeneous Knowledge: Its multimodal knowledge unification decomposes raw inputs into atomic units while preserving structural context and semantic alignment.Figures remain grounded in captions, equations remain linked to definitions, and tables remain connected to explanatory narratives.
2.2.1 Dual-Graph Construction for Multimodal Knowledge: The framework builds complementary cross-modal and text-based knowledge graphs to preserve modality-specific grounding and fine-grained textual semantics.Non-text units become structured graph entities, with multimodal language-model representations generated from local document context.
2.2.2 Graph Fusion and Index Creation: Entity alignment fuses both graphs into a unified knowledge graph, while dense embeddings map entities, relations, and multimodal chunks into a shared retrieval space.The resulting index combines structural knowledge representation with dense vector search.
2.3 Hybrid Retrieval Integration: Hybrid retrieval combines graph-based knowledge navigation with dense similarity matching to provide comprehensive coverage of relevant multimodal knowledge.This design leverages complementary strengths of structural graphs and dense representations for response generation.

3 EVALUATION

RAG-Anything is evaluated on multimodal benchmarks spanning diverse domains, document types, and lengths. It outperforms baselines overall, with especially large gains on long documents and tasks requiring structure-aware localization.

3.2 PERFORMANCE COMPARISON: RAG-Anything achieves superior performance across both DocBench and MMLongBench through unified multimodal processing and dual-graph construction.Unlike LightRAG and MMGraphRAG, it treats text, images, tables, and equations as first-class entities while preserving cross-modal structure.
3.2 PERFORMANCE COMPARISON: RAG-Anything gains 3.4, 9.3, and 7.9 points on MMLongBench documents spanning 11–50, 51–100, and 101–200 pages.The reported improvements are consistent across all listed length categories.
3.3 ARCHITECTURAL VALIDATION WITH ABLATION STUDIES: The chunk-only variant reaches 60.0% accuracy, while removing the reranker yields 62.4% versus 63.4% for the full model.The ablation results attribute the primary gains to graph-based retrieval and cross-modal integration, with reranking providing a smaller refinement.
3.4 CASE STUDIES: A visual-layout graph guides multi-panel queries toward the style-space panel and avoids confusion with the adjacent content-space panel.Panels, axes, legends, captions, and subfigures become linked graph nodes for targeted interpretation.
3.4 CASE STUDIES: A row–column–unit graph retrieves the “Wages and salaries” entry for 2020 as 26,778 million while disambiguating nearby terms.The structure models row headers, year columns, data cells, and units instead of treating the table as linear text.

4 RELATED WORK •

Related work develops graph-based retrieval and multimodal RAG, but existing systems remain fragmented across modality-specific architectures. RAG-Anything addresses this fragmentation with unified cross-modal structural and semantic retrieval.

Graph-based RAG improves retrieval and reasoning by explicitly modeling relationships for long-context and multi-hop queries.
Existing graph-RAG research spans retrieval-efficient graph construction and hierarchical knowledge aggregation approaches.
Current multimodal RAG systems rely on modality-specific architectures, limiting their generalizability across diverse data types.
RAG-Anything counters architectural fragmentation by unifying cross-modal knowledge with structural navigation and semantic matching.

5 CONCLUSION

RAG-Anything proposes a unified graph-based framework for multimodal retrieval and reports strong performance on long-context documents. Its analysis also identifies text-centric bias and rigid spatial processing as challenges for current systems.

RAG-Anything combines dual-graph construction with hybrid structural and semantic retrieval for heterogeneous multimodal documents.
The framework demonstrates superior performance on long-context documents, especially those exceeding 100 pages.
Failure analysis identifies text-centric retrieval bias and rigid spatial processing as major limitations of current multimodal RAG systems.
The analysis motivates adaptive spatial reasoning and layout-aware parsing for complex multimodal documents.

A APPENDIX

The appendix provides supporting materials for the experimental evaluation and implementation details. It includes benchmark statistics, multimodal case studies, and additional implementation documentation.

The appendix documents supporting materials for experimental evaluation and implementation details.
Section A.1 reports dataset statistics for the DocBench and MMLongBench multimodal benchmarks.
Section A.2 presents case studies of structure-aware multimodal content understanding.

A.1 DATASET CHARACTERISTICS AND STATISTICS

DocBench and MMLongBench cover diverse document types and lengths, supporting evaluation across brief news articles and extensive technical or financial documents. Financial reports are especially long in DocBench.

Tables 5 and 6 present document-type distributions and statistics for DocBench and MMLongBench.
Financial reports are the most extensive DocBench category, averaging 192 pages per document.
DocBench includes medium- to long-length legal, governmental, and financial files.
The benchmarks cover document types ranging from concise news articles to extensive technical and financial documentation.

A.2 ADDITIONAL CASE STUDIES

RAG-Anything uses structured representations to reason over visual charts and ambiguous tables. It correctly resolves spatial and structural relationships that challenge baseline approaches.

Visual Reasoning: RAG-Anything identifies “-S-A” as the lowest-accuracy configuration in the Twitter15 chart, unlike baselines that misinterpret spatial relationships.The system represents bars, labels, and legends as connected nodes with explicit semantic relations.
Visual Reasoning: RAG-Anything constructs a structured graph linking chart bars, axis labels, and legends to preserve visual-textual associations.This alignment helps prevent conflation of nearby visual components during retrieval and reasoning.
Tabular Navigation: 0.506 is the highest AUPRC value for Evidence Inference, achieved by “GloVe + LSTM – Attention” under RAG-Anything’s table navigation.Relational parsing preserves dataset context and explicit row-of and column-of relationships.
Tabular Navigation: The framework parses complex tables into relational graphs so headers and cells retain hierarchical constraints during navigation.This addresses repeated row labels and structural ambiguity across datasets in a single table.

A.3 CONTEXT-AWARE MULTIMODAL PROMPTING

The framework uses context-aware prompts to analyze images, tables, and equations in structured JSON outputs. These prompts connect multimodal content to surrounding context and guide detailed semantic extraction.

Context-Aware Prompting: Three prompts orchestrate context-aware analysis of visual, tabular, and mathematical content with JSON-formatted outputs.They guide comprehensive extraction while maintaining alignment with surrounding information.
Vision Analysis: The vision prompt captures image composition, object relationships, visual attributes, technical components, and connections to accompanying text.Its goal is contextual interpretation rather than superficial visual description.
Table Analysis: The table prompt decomposes structure, column semantics, critical values, statistical patterns, and contextual relevance with numerical accuracy requirements.It preserves key indicators while reducing ambiguous generalizations.
Equation Analysis: The equation prompt emphasizes variable definitions, operational logic, theoretical foundations, inter-formula relationships, and practical applications.Mathematical expressions are interpreted semantically and integrated into broader arguments.

A.4 ACCURACY EVALUATION PROMPT DESIGN

Accuracy evaluation uses a standardized prompt to assess generated responses consistently across domains and question types. GPT-4o-mini performs all evaluations using binary classifications and concise justifications.

Evaluation Protocol: The standardized prompt prioritizes factual correctness over style and produces binary accuracy classifications with concise analytical justifications.The evaluation covers generated responses across multiple domains.
Evaluation Protocol: GPT-4o-mini conducts all accuracy evaluations using consistent standards across diverse question categories and specialized domains.This supports comparable assessment throughout the experimental framework.

A.5 CHALLENGES AND FUTURE DIRECTIONS FOR MULTI-MODAL RAG

Failure analysis identifies text-centric retrieval and rigid spatial processing as systemic weaknesses in multimodal RAG. The cases motivate adaptive layout parsing and visual processing for greater robustness.

Cross-Modal Misalignment: Current systems favor textual sources when visual content lacks exact keyword matches, even when queries explicitly prioritize visual information.Topically related text may differ from images in granularity and specificity, introducing misleading noise.
Cross-Modal Misalignment: All evaluated methods fail in a cross-modal noise case, retrieving misleading textual evidence instead of the correct structured visual information.The failure occurs despite access to the necessary information and reflects cross-modal misalignment.
Structural Reasoning: Rigid top-to-bottom and left-to-right scanning fails when documents require non-conventional spatial strategies such as column-wise table interpretation.Professional documents commonly contain structural variations requiring adaptive spatial reasoning.
Structural Reasoning: A reverse-order visual integration requirement exposes static spatial processing that can produce incomplete or actively misleading information.Misalignment between processing patterns and document structure may lead to confident but incorrect conclusions.
Structural Reasoning: All methods fail on an ambiguous table where missing cell boundaries and merged columns make structural parsing and data extraction unreliable.The case exposes brittleness when document layouts deviate from clean structured formats.
Future Directions: Robust multimodal RAG should combine layout-aware parsing with visual processing that uses spatial relationships and contextual design cues.These directions are presented as ways to improve resilience to structural irregularities and noise.