Source-linked AI summary
ChartWalker: Benchmarking the Cross-Chart RAG Task with Hierarchical Knowledge Graphs
Ning Tang, Chenghan Xie, Hanyang Yuan, Yi Li, Renhong Huang, Qian Kou, Xiaofeng Shi, Hua Zhou, Jiarong Xu
TL;DR
Existing cross-chart RAG benchmarks inadequately capture charts’ multimodal structure and coherent multi-hop reasoning. ChartWalker addresses this with hierarchical knowledge graphs and structure-aware path sampling, and evaluations show current systems struggle on complex multi-chart analysis.
Problem
Existing benchmarks inadequately capture charts’ multimodal structure and the logically coherent reasoning required by realistic cross-chart queries.
Method
ChartWalker builds hierarchical chart knowledge graphs and uses structure-aware sampling to synthesize coherent multi-hop reasoning paths for grounded QA.
Results
Current vision-language models struggle with complex multi-chart analysis, with accuracy below 30% for most complex reasoning queries.
Takeaways & Limitations
ChartWalker-Bench provides a rigorous evaluation foundation for multimodal RAG systems targeting complex chart analysis.
Takeaways & Limitations
The hierarchy reflects source-chart informativeness rather than a semantic or linguistic hierarchy, so identical entities may occur in different layers.
Abstract
from arXiv · showhide
Cross-Chart Retrieval-Augmented Generation (RAG) is critical for complex multi-modal analytical tasks in scientific, business, and political domains. However, existing benchmarks either focus on tables, which are well-structured and textualized, or generate cross-chart questions by simply extracting key points, which often induces lexical overlap between queries and evidence and yields logically inconsistent reasoning chains. To address this, we introduce ChartWalker, a novel framework for constructing challenging cross-chart RAG tasks. ChartWalker features a hierarchical knowledge graph construction method tailored to charts, which organizes entities and relations by granularity to preserve analytical structure. We then propose a structure-aware sampling algorithm that synthesizes semantically coherent, multi-hop reasoning paths, enabling explicit control over query difficulty and granularity for QA generation. Built with this framework, we release ChartWalker-Bench, a comprehensive benchmark spanning diverse domains and cross-chart query types. Extensive evaluations across major RAG paradigms reveal significant performance gaps, underscoring the benchmark's difficulty and utility. Furthermore, we provide ChartWalker-Agent, an agentic baseline to facilitate analysis and inspire future system design.
1. Inroduction
ChartWalker addresses the lack of benchmarks capturing charts’ multimodal structure and realistic cross-chart reasoning by constructing hierarchical knowledge graphs and sampling grounded multi-hop paths. It also introduces ChartWalker-Bench and ChartWalker-Agent, with experiments showing substantial difficulty for current RAG systems.
- Motivation: Charts are information-dense and weakly structured, yet real-world analysis often requires synthesizing evidence across multiple charts.Charts support quantitative visualization across science, business, journalism, and policy, while cross-chart questions may connect trends such as GDP growth and inflation.
- Limitations: Existing benchmarks inadequately capture charts’ multimodal nature and the reasoning structure required by realistic cross-chart queries.Prior work largely focuses on tables with explicit entities and relations, while existing chart benchmarks have important construction limitations.
- ChartWalker: ChartWalker constructs hierarchical knowledge graphs that organize chart entities and relations by information granularity, then uses structure-aware sampling to synthesize semantically coherent reasoning paths.The framework explicitly exposes multi-granular chart structure and supports grounded, multi-hop question generation.
- ChartWalker-Bench: ChartWalker-Bench contains 564 multi-hop QA instances across 4 query types, with annotations grounded on explicit reasoning chains.The benchmark is evaluated across major RAG paradigms and different VLM generators.
2. Related Work
Related work spans chart question answering, cross-chart RAG benchmarking, and multi-hop question generation. Existing approaches often rely on structured table contexts or semantic similarity, motivating more coherent cross-chart reasoning benchmarks.
- Chart RAG Benchmark: Early chart QA focused on tables with given contexts and relatively simple operations, while later work addressed complex chart reasoning as VLM capabilities advanced.Tables, graphs, and diagrams are treated as chart instances.
- Chart RAG Benchmark: ChartMRAG was among the first cross-chart RAG benchmarks, but its semantic-similarity reliance can produce incoherent reasoning paths.This limits its reliability for evaluating cross-chart reasoning.
- Multi-hop Question Generation (MHQG): Multi-hop question generation synthesizes questions requiring multi-step reasoning across multiple contexts, with early methods using explicit structured representations to scaffold reasoning.Knowledge-graph path sampling has been used for difficulty-controllable generation, with hop count serving as a proxy for reasoning difficulty.
3. ChartWalker Benchmark
ChartWalker-Bench constructs chart RAG evaluations with hierarchical knowledge graphs and structure-aware path sampling to produce diverse-granularity queries with coherent, verifiable rationales. Its questions require multi-source, multi-branch reasoning, with the Complex Reasoning subset reaching the highest evidence demands.
- Benchmark construction: ChartWalker-Bench provides chart query–answer pairs spanning diverse granularities with logically coherent, verifiable rationales for systematic evaluation.The benchmark construction explicitly links entities within and across charts and uses path sampling for QA generation.
- Knowledge graph: The hierarchical knowledge graph links chart entities and relations across heterogeneous charts while preserving information granularity.Nodes include titles, legends, and individual units; levels distinguish coarser globally informative entities from finer-grained datapoints.
- Knowledge graph: Identical entities are merged within the same granularity level across charts, while occurrences at different levels remain separate because levels reflect source-chart informativeness.The level is not a semantic or linguistic hierarchy, so cross-level reconciliation is not performed.
- Path sampling: Structure-aware path sampling constrains next hops by semantic topic coherence and entity levels to limit semantic drift and control query granularity.The policy can favor upward-level transitions for finer-grained queries or shallower levels for coarser-grained queries, terminating at the hop budget or a dead end.
- Benchmark complexity: 2.30 evidence sources and 2.81 reasoning branches are used per question on average, while Complex Reasoning reaches 3.12 source charts and 5.14 reasoning hops.This subset stresses retrieval of the right charts under distraction and integration of multiple evidence pieces.
4. ChartWalker Agent
ChartWalker Agent is introduced as an agentic baseline for heterogeneous cross-chart reasoning, where queries require different retrieval granularities and reasoning processes. It iteratively explores a chart-derived knowledge graph, retrieves evidence, and optimizes decisions through reward-driven training.
- Motivation: Cross-chart queries require varying evidence granularities and distinct reasoning processes, limiting static retrieval pipelines on complex multi-hop reasoning.This heterogeneity is identified as a key challenge beyond chart perception.
- Agent Design: ChartWalker Agent treats retrieval as sequential decision-making to support iterative evidence acquisition and dynamic retrieval strategies.The design is motivated by agentic RAG approaches that observe the corpus, maintain searching memory, and adapt retrieval over time.
- Environment and Action Space: The agent operates on a standard knowledge graph rebuilt from chart summaries and their entity relations to avoid reusing the benchmark-construction hierarchy.Each chart summary is treated as a textual passage during knowledge-graph construction.
- Training Objective and Optimization: At each turn, the agent observes graph context and retrieved evidence, takes an exploratory action, and receives a scalar environmental reward.Its policy is parameterized by a vision-language model that autoregressively outputs token sequences as actions.
- Training Objective and Optimization: The training objective is to maximize expected discounted return during iterative knowledge-graph exploration.The agent’s action policy is implemented by a vision-language model.
5. Experiments
Experiments show that ChartWalker-Bench remains difficult for both retrieval and answer generation, while multimodal representations, graph-aware retrieval, and stronger VLMs provide complementary gains. ChartWalker-Agent further shows that retrieval-budget effects are non-monotonic for lightweight models.
- Overall difficulty: The benchmark remains challenging, with best retrieval R@10 ≈72 and best overall Cor@10 ∼65; Complex Reasoning reaches only ≈51.These results indicate persistent bottlenecks in limited-budget retrieval, multi-source retrieval, and evidence composition.
- Retrieval: VL-Embedding achieves the strongest average recall, 53.90/71.87, and ranks first across most categories.The reported gains are attributed to aligning query text with chart visuals in a shared embedding space.
- Retrieval: HippoRAG outperforms the text-only retriever by +8.66 R@5 and +12.37 R@10 overall, especially on Fact Check and Manipulation.Graph propagation aggregates multi-hop evidence and recovers supporting sources that single-pass embedding search may miss.
- Generation: Qwen3-VL-32B consistently outperforms Qwen3-VL-8B, while GPT-4o with VL-Embedding achieves the best overall scores.HippoRAG remains best at Cor@10 for Analysis (70.66) and Complex Reasoning (51.38), even with GPT-4o.
- ChartWalker-Agent: For 3B ChartWalker-Agent models, increasing retrieval from k=5 to k=10 can reduce correctness, with HippoRAG dropping from 31.74 to 29.79.More charts can exceed multi-image context or visual-token limits and introduce distractors.
6. Conclusion
ChartWalker introduces a benchmark and framework for challenging Cross-Chart RAG through hierarchical knowledge graphs and structure-aware sampling. Evaluations show that current Vision-Language Models struggle with multi-chart analysis, motivating the proposed ChartWalker-Agent.
- Contributions: ChartWalker is a framework and benchmark for Cross-Chart RAG.It is designed to evaluate complex cross-chart analytical tasks.
- Contributions: Hierarchical knowledge graphs and structure-aware sampling generate complex, multi-hop reasoning paths.These mechanisms are intended to challenge existing systems.
- Findings: Current Vision-Language Models struggle with multi-chart analysis on ChartWalker-Bench, exposing limitations of static retrieval.The evaluation motivates approaches beyond static retrieval.
- Agentic Baseline: ChartWalker-Agent is proposed to help bridge the gap revealed by these evaluation results.The paper presents it as an agentic approach following the benchmark findings.
Impact Statement
The paper aims to advance machine learning and acknowledges that its work may have many societal consequences. However, it does not identify any consequences as requiring specific emphasis.
- The paper’s stated goal is to advance the field of machine learning.
- The authors acknowledge that their work may have many potential societal consequences.
- The impact statement does not specifically highlight any societal consequence.
A. Appendix. · A.1. Notations
Appendix A.1 provides a summary table of the notations and symbols used throughout ChartWalker.
- A.1. Notations: Table 8 serves as a reference for the symbols used in ChartWalker.The passage describes the table as a summary of notations and symbols.
- A.1. Notations: The appendix includes a dedicated notations subsection for ChartWalker.The passage is associated with Appendix A.1, titled “Notations.”
- A.1. Notations: Appendix A.1 documents the symbols used by the ChartWalker framework.This point restates the table’s stated purpose without adding external content.
- A. Appendix.: The provided appendix material is limited to a reference summary of ChartWalker notation and symbols.No individual notation definitions are included in the supplied passage.
Problem Formulation
The problem formulation centers on a Hierarchical Knowledge Graph and a Path Sampling & QA Synthesis component.
- The formulation includes a Hierarchical Knowledge Graph.
- These are the two named components of the problem formulation.
- It also includes Path Sampling & QA Synthesis.
A.2. More experiment settings
The experiments compare sparse, dense textual, vision-language, and graph-based retrieval baselines. Retrieval configurations specify ranking parameters, seed selection, neighborhood expansion, and hierarchical multi-hop traversal.
- Baseline Details: The baselines include BM25, dense textual embedding, vision-language embedding, and RagAnything in local and mixed modes.BM25 uses term matching; dense textual embedding textualizes visual sources; vision-language embedding unifies textual and visual features; RagAnything provides graph-based support.
- Retrieval hyperparameters: BM25 uses Okapi parameters k1=1.5 and b=0.75, while RagAnything Local retrieves top-10 seed entities with one-hop expansion.RagAnything Mix instead starts with top-10 seed candidates and expands to the top-8 neighbors per seed.
- Retrieval hyperparameters: HippoRAG uses hierarchical Personalized PageRank with damping factor=0.5 to propagate relevance and traverse multiple hops to relevant passages.The configuration explicitly enables multi-hop traversal through hierarchical relevance propagation.
A.3. Prompt Templates … A.5. Showcase
The appendix specifies prompts for constructing hierarchical chart knowledge graphs, generating multi-chart QA, and navigating the resulting multimodal graph. It also describes a turn-based agent environment and showcases manipulation, complex reasoning, fact-checking, and analysis queries.
- A.3. Prompt Templates: The entity-extraction prompt builds an adaptively layered knowledge graph from chart structure, with deeper levels recursively representing finer-grained dimensions and attributes.Level 1 captures core topics and primary comparisons; subsequent levels capture subcategories and supporting attributes.
- A.3. Prompt Templates: The prompt requires semantic, disambiguated entities and typed intra- and inter-level relationships, returning them in a JSON object with entities and relationships fields.Entities include uniqueness qualifiers, while relationships connect higher-level entities to their children and use required relation types.
- Unified Hierarchical Entity Extraction & QA Generation Prompt: The unified multi-chart QA template shares common context and output formatting while branching into four modules with task-specific logic, constraints, and paraphrasing requirements.Its inputs include reasoning paths, available sources, a chart index, and textual evidence.
- A.4. Agent Environment: ChartWalker models visual search over the global chart knowledge graph as a POMDP in which each turn produces an exploration action and scalar reward.The agent receives the current graph context and retrieved evidence as its observation.
- A.4. Agent Environment: The non-concatenated rollout conditions each turn only on its current prompt context, avoiding context-window overflow and ultra-long token-axis credit assignment.PPO is computed per turn, with turn-level advantages broadcast to response tokens and turn-level value regression.
- ChartWalker Agent: Navigation & Reasoning System Prompt: The agent navigation prompt directs the system to identify required information, traverse relevant entities, extract concrete evidence, and stop when sufficient information is collected.Actions that do not fill missing information are discouraged, and the prompt organizes interaction into start-entity selection, iterative navigation, and termination.
- A.5. Showcase: The showcase includes a manipulation example avoiding a reported logical inconsistency, a top-k set-membership query, a Peru percentage comparison, and cross-attitudinal correlation analysis.These examples use sampled knowledge-graph paths linking chart entities, values, and contextual evidence.