When Graph Data Meets Multimodal: A New Paradigm for Graph Understanding and Reasoning

1 Introduction

• Graph neural networks capture complex graph features, but face scalability and broader-processing challenges, and each graph type requires a separately trained model.Graph types such as knowledge graphs and mind maps have distinct structures; traditional matrix-based and classical methods struggle with diverse, complex graphs [16].Source: §1 · p. 1
• Integrating graph structure with the sequential nature of large language models is difficult, and modifying language models to incorporate GNN features can require complex retraining costing several hundred thousand dollars.The integration challenge is discussed with references [28, 4].Source: §1 · pp. 1–2
• The proposed paradigm processes graph data through image encoding and multimodal integration, enabling comprehension in an instruction–response format.The study investigates GPT-4V in this setting.Source: §1 · p. 2
• The initial exploration covers knowledge graphs, flowcharts, mind maps, route maps, and Gantt charts, evaluating images through command generation and instruction adherence.Adherence evaluation considers command complexity, image noise, and information density.Source: §1 · pp. 2–3
• GPT-4V shows some image-understanding and reasoning capability, but struggles with Chinese OCR and specialized formats such as Gantt charts and route maps.The evaluation also describes challenges in complex reasoning tasks.Source: §1 · p. 3

2.1.1 Graph Image Acquisition

• The dataset contains 156 knowledge graphs, 242 route maps, 157 flowcharts, 40 Gantt charts, and 31 mind maps, with approximately three Chinese-annotated images per English-annotated image.Images were gathered through keyword and reverse-image searches, then filtered for clarity and relevance.Source: §2.1.1 · p. 3
• GPT-4 receives graph images and generates alternating questions and answers, represented as {Q1, A1, Q2, A2, ...}.Human evaluators manually judge instruction validity and the accuracy of responses to valid instructions.Source: §2.1.1 · p. 3, §2.1.2 · p. 4

2.1.2 What Ability Are We Focus On?

• Response evaluation covers simple and medium instruction following, complex instruction following, multi-hop reasoning, robustness to noise, information density, and graph type.Complex instructions can require higher-order comprehension while still eliciting simple answers; multi-hop reasoning links several inferential steps and usually produces more elaborate responses.Source: §2.1.2 · pp. 3–4

2.2 Prompt Design

• The initial prompt, inspired by [29], asks GPT-4V to act as a knowledge-graph expert, identify the image correctly, and produce correct questions and answers.The initial prompt often elicited short, straightforward questions without complex reasoning or multi-hop inference.Source: §2.2 · p. 4
• The refined prompt asks questions to focus on specific nodes or edges and range from simple to complex.The prompt was adapted for Chinese-content images and customized by replacing “knowledge graph” with the relevant graph type.Source: §2.2 · p. 4
• Focusing on graph elements and varying question complexity significantly improved the quality and depth of generated questions.The resulting questions ranged from simple factual inquiries to more complex inferential ones.Source: §2.2 · p. 4
• Figure 6 and Figure 7 show generated instruction-and-response examples for an English knowledge graph and a Chinese flowchart, respectively.Both examples follow prompts tailored to the graph type and language.Source: §2.2 · p. 4

2.3 Instruction Generation Ability

• GPT-4 generated instructions for 50 images, balanced between Chinese and English annotations, typically producing 3–9 questions per image across simple, medium, and complex levels.The model assigned complexity labels in its responses.Source: §2.3 · p. 4
• Figure 6 shows instructions and answers generated for an English-annotated image following the user prompt.The example illustrates the instruction-generation format.Source: Figure 6 · p. 5

2.3.1 Relevance Assessment

• Instructions were invalid if unrelated to the image’s theme or if they referred to nodes or edges absent from the image.The study assessed relevance by checking generated instructions against the corresponding image.Source: §2.3.1 · p. 5
• The reported validity-assessment distribution is presented in Table 1.The supplied passage identifies the table’s scope but does not provide its values.Source: Table 1 · p. 5
• Instruction-generation errors were particularly high for Chinese-annotated images and quite low for English images, prompting further Chinese OCR testing.The authors suspected Chinese OCR problems but described that explanation as speculation at this stage.Source: §2.3.1 · p. 5

2.3.2 OCR Performance Test

• GPT-4V’s Chinese OCR accuracy was approximately 29.8% in a test using one image from each of five graph categories.Accuracy was calculated from correct node and edge recognitions relative to total recognitions; errors included similar-looking characters and inexplicable text substitutions.Source: Table 2 · p. 6, §2.3.2 · p. 6

2.4 Simple and Medium Instruction Following Ability

• The analysis sampled ten images containing 55 valid simple or intermediate instructions after finding little difference between those difficulty levels.The paper reports that the comparison supported merging the categories for collective analysis.Source: §2.4 · p. 7
• Table 3 compares response accuracy for simple and medium instructions, whose rates were similar in the analysis.This similarity supported combining the two complexity categories.Source: Table 3 · p. 7, §2.4 · p. 7
• GPT-4V struggled to align Gantt-chart bars with task times, leading to errors in task start times and overlaps.Other errors included misrecognizing node names or their meanings and omitting or spuriously adding subnodes.Source: §2.4 · p. 7

• Complex questions exposed errors in recognizing node values and judging relationships across multiple nodes and edges.Source: §2.5 · p. 7
• Complex-instruction accuracy was a little lower than accuracy on simple and medium instructions, but remained satisfactory.Source: Table 4 · p. 7
• On the route map, GPT-4V misplaced day 7’s starting point and confused directed-edge directions during complex reasoning.Source: §2.5 · p. 7

• The evaluation sampled ten images containing multi-hop questions and assessed 19 instructions.Source: §2.6 · p. 7
• 63.2% (12 out of 19) accuracy on multi-hop instructions was nearly 10 percentage points below complex-instruction accuracy.Source: Table 5 · p. 7, §2.6 · p. 7
• The reasoning analysis focused on pattern-recognition errors and errors understanding logical or mathematical relationships in graphics.Source: §2.6 · p. 7

• Blurry images produced an increase in invalid instructions compared with images without noise across five images and 40 instructions.Source: §2.7 · p. 8, Table 6 · p. 9
• Limited pattern recognition led GPT-4V to generate invalid questions, including miscounting branching nodes or asking about nonexistent nodes.Source: §2.7 · p. 8
• Response accuracy declined across 30 valid instructions, and the paper identifies image clarity as foundational to strong performance.Source: §2.7 · p. 8

• The comparison used ten images, five per density category, with 25 instructions evaluated for each category.Source: §2.8 · p. 9
• Higher error rates were observed on dense images and attributed to weaker pattern recognition and relationship understanding.Source: §2.8 · p. 9, Table 8 · p. 10
• Dense nodes and edges can confuse pattern recognition, while varied information can make intricate graph relationships harder to comprehend.Source: §2.8 · pp. 9–10

2.9 Ablation Study on Different Types of Graphs

• 211 of 270 answers were correct across ten noise-free pictures per graph category, with performance varying significantly by image type.Answers to valid instructions were manually evaluated.Source: §2.9 · p. 10
• GPT-4V struggled to determine overlapping task times and individual durations in Gantt charts, with difficulty aligning bars to timelines.These issues led to incorrect answers.Source: §2.9 · p. 10
• Mind-map instruction generation was error-prone when estimating a node’s sub-branches, while route planning produced unreachable destinations and impractical paths.Impractical routes included retracing or impassable paths.Source: §2.9 · p. 10

3 Conclusion and Future Work

• The study reports GPT-4V proficiency on varied graph tasks, but identifies Chinese OCR and specialized formats such as Gantt charts and route maps as challenging.The conclusion calls for improved non-Latin OCR and better understanding of complex graphical structures.Source: §3 · p. 10
• The conclusion presents the study as a basis for future research on AI multimodal interaction and reasoning.Source: §3 · p. 11
• The authors initiated annotation of a medium-scale graph dataset containing images, instructions, and responses to address the lack of graph data in open-source multimodal datasets.The dataset is intended to improve multimodal models’ graph-data processing.Source: §3 · p. 11
• The authors propose using annotated data to fine-tune open-source models, with emphasis on Chinese graph data and weaker graph types such as Gantt charts and route maps.Their analysis reports that open-source models, particularly in Chinese, are less effective than GPT-4V.Source: §3 · p. 11
• The authors call for stronger Chinese OCR and discuss aligning Chinese text in images with textual counterparts, including training with existing Chinese OCR datasets.Source: §3 · p. 11
• For graphs with thousands of nodes and edges, the authors propose sequential subgraphs processed with multimodal methods, requiring models to reason jointly across multiple images.They describe this as a future capability to explore for more detailed processing of large graphs.Source: §3 · pp. 11–12