PaperlayerView exact revision on arXiv

Source-linked AI summary

Aligning Vision to Language: Annotation-Free Multimodal Knowledge Graph Construction for Enhanced LLMs Reasoning

Junming Liu, Siyuan Meng, Yanting Gao, Song Mao, Pinlong Cai, Guohang Yan, Yirong Chen, Zilin Bian, Ding Wang, Botian Shi

arXiv:2503.12972v3cs.CVcs.AI

TL;DR

Multimodal reasoning in LLMs faces incomplete knowledge and hallucinations, while text KGs cannot bridge visual and textual semantics and MMKG construction is hindered by annotation demands and noisy relations. VaLiK builds annotation-free MMKGs by aligning image features with text and verifying cross-modal consistency; experiments report state-of-the-art reasoning results alongside storage savings and preserved entity-to-image links.

  • Problem

    LLM multimodal reasoning faces incomplete knowledge and hallucinations, while text KGs cannot bridge modalities and MMKG construction is constrained by annotation scarcity and noisy visual relations.

    Source: §1 · pp. 1–2

  • Method

    VaLiK cascades pretrained VLMs to generate image-specific descriptions, verifies cross-modal similarity to filter noise, and structures the refined descriptions into MMKGs without manual image captions.

    Source: §1 · p. 2, §3.2 · p. 4, §3.3 · p. 5

  • Results

    VaLiK-augmented LLMs achieve state-of-the-art multimodal reasoning across benchmarks; on ScienceQA, Qwen2.5-72B achieves state-of-the-art performance on 62.5% of subtasks.

    Source: §4.2 · p. 7, §5 · p. 8

  • Takeaways & Limitations

    VaLiK's compressed graphs require less storage while preserving direct entity-to-image links, and its modular design supports adaptation across domains.

    Source: §1 · p. 2

Abstract

from arXiv · show

Multimodal reasoning in Large Language Models (LLMs) struggles with incomplete knowledge and hallucination artifacts, challenges that textual Knowledge Graphs (KGs) only partially mitigate due to their modality isolation. While Multimodal Knowledge Graphs (MMKGs) promise enhanced cross-modal understanding, their practical construction is impeded by semantic narrowness of manual text annotations and inherent noise in visual-semantic entity linkages. In this paper, we propose Vision-align-to-Language integrated Knowledge Graph (VaLiK), a novel approach for constructing MMKGs that enhances LLMs reasoning through cross-modal information supplementation. Specifically, we cascade pre-trained Vision-Language Models (VLMs) to align image features with text, transforming them into descriptions that encapsulate image-specific information. Furthermore, we developed a cross-modal similarity verification mechanism to quantify semantic consistency, effectively filtering out noise introduced during feature alignment. Even without manually annotated image captions, the refined descriptions alone suffice to construct the MMKG. Compared to conventional MMKGs construction paradigms, our approach achieves substantial storage efficiency gains while maintaining direct entity-to-image linkage capability. Experimental results on multimodal reasoning tasks demonstrate that LLMs augmented with VaLiK outperform previous state-of-the-art models. Our code is published at https://github.com/Wings-Of-Disaster/VaLiK.

1. Introduction

VaLiK addresses the annotation and semantic-reliability obstacles to constructing MMKGs by aligning image features with text, verifying cross-modal consistency, and organizing the refined descriptions into a graph. Experiments report stronger multimodal reasoning with less storage while retaining entity-to-image links.

1. Introduction

  • MMKG construction is constrained by scarce fine-grained entity-image corpora and visual relation detectors that produce superficial or hallucinated connections.The paper links these issues to scalability and graph integrity, respectively.Source: §1 · p. 2
  • VaLiK cascades pretrained VLMs to create image-specific descriptions, filters inconsistent cross-modal relations, and uses LLM-driven structuring to build MMKGs without manual image captions.The approach also extracts KG-compatible semantic relationships beyond predefined label taxonomies.Source: §1 · p. 2
  • Feature-aligned VLM descriptions can introduce redundant and inaccurate relationship patterns that require filtering.The paper illustrates this issue in Figure 2.Source: Figure 2 · p. 2
  • VaLiK-constructed MMKGs improve LLM multimodal reasoning while requiring substantially less storage and retaining direct entity-to-image links.The introduction identifies evaluation across multimodal classification and question answering benchmarks.Source: §1 · p. 2
  • VaLiK is presented as an end-to-end, annotation-free, zero-shot, storage-efficient framework with modularity for adapting to specialized domains.The stated contributions also include knowledge distillation that reduces storage while maintaining semantic integrity.Source: §1 · p. 2

2. Related Work

Related work presents MMKGs as a way to integrate visual and textual information, while noting that construction often depends on extensive annotation. Knowledge-augmented multimodal learning and multimodal LLMs improve cross-modal capabilities, but face domain-mismatch and hallucination concerns.

2.1. Multimodal Knowledge Graphs

  • MMKGs link entities to visual or textual data to support multimodal reasoning, but existing construction approaches can require extensively annotated datasets.Prior work addresses continual MMKG construction, robotic-manipulation knowledge, and quad-modal graph grounding.Source: §2.1 · pp. 2–3

2.2. Knowledge-Augmented Multimodal Learning

  • Knowledge-augmented multimodal methods improve cross-modal grounding, but preconstructed generic MMKG schemas can mismatch task-specific reasoning patterns.Examples include dual-KG adaptation, contrastive multi-relational encoding, and task-specific MMKG construction.Source: §2.2 · p. 3

2.3. Multimodal Large Language Models

  • Multimodal LLMs connect visual and textual representations through alignment and interaction mechanisms, yet remain prone to hallucinations and do not acquire new knowledge through integration alone.VaLiK instead constructs MMKGs with multimodal models and uses them to augment multimodal reasoning.Source: §2.3 · p. 3

3. Method

VaLiK turns image content into text descriptions, verifies their alignment with images, and uses the refined descriptions to construct an MMKG for LLM reasoning. Its method combines cascaded VLM generation, window-level filtering, and graph-based retrieval.

3.1. CoE-based Visual to Language Modeling

  • Cascaded CoE-based VLMs process images and preceding experts’ outputs to produce hierarchical, image-specific textual descriptions without requiring fine-grained training data.The pipeline uses pretrained VLMs to extract visual information, align image features with text, and autoregressively generate descriptions.Source: §3 · p. 3, §3.1 · pp. 3–4

3.2. Cross-Modal Similarity Verification

  • A sliding-window verification mechanism scores caption segments for cross-modal similarity and filters out low-similarity windows as noise.The window size can adapt dynamically to natural sentence segmentation.Source: §3.2 · pp. 4–5

3.3. MMKG Construction for Enhanced Reasoning

  • An LLM constructs the MMKG from refined visual descriptions, optionally concatenated with external text, and represents knowledge as entity-relation triplets.The graph can include entities from images or external text, with relations linking head and tail entities.Source: §3.3 · p. 5
  • At inference, VaLiK retrieves query-relevant graph subgraphs, and image storage locations let the MMKG link entities directly to visual data.The retrieved triplets are used to support reasoning, while visual associations enable text-only LLMs to perform multimodal reasoning.Source: §3.3 · p. 5

4. Experiment

VaLiK is evaluated on CrisisMMD classification and ScienceQA question answering across LLM architectures and knowledge-graph configurations. It improves multimodal reasoning, with gains varying by model and dataset, while ablations show that its components have context-sensitive effects.

4.1. Setups

  • VaLiK is evaluated on CrisisMMD disaster-post classification and ScienceQA multimodal science questions, comparing text-only LLMs, VLMs, and KG-assisted LLMs across these benchmarks.Implementation uses Qwen2.5-7B for LightRAG hybrid retrieval and DeepSeek-R1-70B for graph construction; KG comparisons include LightRAG, Visual Genome, and Mmkg.Source: §4.1 · pp. 5–6

4.2. Main Results

  • On CrisisMMD, VaLiK improves Qwen2.5-7B accuracy by 4.41% with image-only KGs and 4.90% with text-image KGs; on ScienceQA, Qwen2.5-72B gains 6.4% average accuracy and reaches SOTA on 62.5% of subtasks.The CrisisMMD gains exceed LightRAG’s 1.22% gain; ScienceQA results cover the benchmark’s question, context, and educational-stage categories.Source: §4.2 · pp. 6–7, Table 1 · p. 6, Table 2 · p. 6
  • VaLiK-enhanced Qwen2.5-7B matches native Qwen2.5-72B on CrisisMMD, while Qwen2.5-32B and 72B gain 2.0%–2.5%; VaLiK requires no task-specific adaptation.The authors note that models with more substantial prior knowledge benefit less from external knowledge augmentation.Source: §4.2 · pp. 6–7, Table 1 · p. 6
  • For LLaVA-34B, VaLiK gains 2.41% with image-only and 3.59% with text-image KGs; Qwen2-VL-72B-Instruct gains 1.77% and 2.23%, respectively.The paper reports smaller VaLiK gains for VLMs than for text-only LLMs, whose visual inputs are not inherently available at inference.Source: §4.2 · pp. 6–7, Table 2 · p. 6
  • On ScienceQA, text-only KGs contain 8× denser structured knowledge than image-only KGs, yet image-only KG augmentation raises Qwen2.5-7B accuracy by 4.16%.VaLiK’s MMKG uses 489MB, compared with 739MB for Visual Genome’s scene graph component, and supports reasoning from textual KG descriptions without raw images.Source: §4.2 · p. 7

4.3. Ablation Study

  • The CoE-based VLM module yields average accuracy gains of +3.05% on CrisisMMD and +4.63% on ScienceQA, while similarity verification helps CrisisMMD but slightly degrades ScienceQA image-only reasoning.The authors attribute the difference to more redundant CrisisMMD captions and the risk of over-removing semantics from ScienceQA’s shorter descriptions.Source: §4.3 · p. 8, Table 4 · p. 8, Table 5 · p. 8

4.4. Further Analysis

  • BLIP-2 produces the fewest descriptions but best extracts critical image information and entity relations; adding VLMs brings diminishing returns on current images.The authors use BLIP-2 as the primary model and LLaVA or Qwen2-VL as secondary or tertiary models; Figure 4 presents the quantity analysis on CrisisMMD.Source: §4.4 · p. 8, Figure 4 · p. 8
  • VaLiK is reported as more cost-effective than manual annotation or LLM fine-tuning.Computational-cost details are provided in Appendix F.Source: §4.4 · p. 8

5. Conclusion

VaLiK constructs MMKGs by converting images into structured knowledge with pretrained VLMs and cross-modal verification. Its graphs are reported to improve multimodal reasoning with minimal storage overhead.

5. Conclusion

  • VaLiK combines a cascade of pretrained VLMs with cross-modal verification to convert images into structured knowledge while filtering alignment noise.The conclusion describes the framework as modular and adaptable across domains.Source: §5 · p. 8
  • Experiments on multimodal reasoning benchmarks show SOTA performance for LLMs augmented with VaLiK.The resulting graphs are also described as having minimal storage overhead.Source: §5 · p. 8
  • VaLiK is presented as a scalable approach to autonomous knowledge synthesis that efficiently integrates visual and textual data.Source: §5 · p. 8

A. Cross-Modal Reasoning Failures in Textual KGs

Textual KGs can leave visual-textual ambiguities unresolved, while MMKGs link entities across modalities so LLMs can retrieve and reason over fused evidence.

A. Cross-Modal Reasoning Failures in Textual KGs

B. Case Studies on Manual Annotation Over-heads

Manual entity-image annotation is labor-intensive, and coarse tagging can omit the fine-grained attributes and relationships needed for compositional reasoning.

B. Case Studies on Manual Annotation Over-heads

  • Flickr30K-Entity requires labor-intensive manual annotation at scale, while its common tagging paradigm often identifies only coarse-grained entities.The passage says this sparsity propagates to trained models, which consequently fail to capture compositional semantics for complex reasoning.Source: B. Case Studies on Manual Annotation… · pp. 14–15
  • The passage describes annotations as neglecting fine-grained attributes and contextual relationships, limiting the semantic enrichment available to trained models.Source: B. Case Studies on Manual Annotation… · pp. 14–15
  • ScienceQA [48] examples involve multimodal question answering that requires joint reasoning over textual prompts and visual evidence.Source: Figure 7 · p. 15

C. Case Studies on Visual Specificity Deficits in VLM-Generated Captions

The BLIP-2 example illustrates how a coarse caption can omit visual details relevant to multimodal reasoning, unlike the more specific human annotation.

C. Case Studies on Visual Specificity Deficits in VLM-Generated Captions

D. Retrieval Strategy in MMKG Construction

VaLiK uses hybrid LightRAG retrieval to support MMKG construction and selects the sensitivity threshold empirically. Threshold choice can be guided by caption length, but pruning is especially sensitive near some threshold values.

D. Retrieval Strategy in MMKG Construction

  • Hybrid LightRAG retrieval balances local cues with global knowledge, while keyword-guided chunking and chunk-level vector retrieval expand accessible context.Qwen2.5-7B was selected for retrieval; 32B and 72B models improved performance by 1%-5% but significantly increased graph construction time.Source: D. Retrieval Strategy in MMKG… · p. 15
  • The MMKG-generation procedure combines the refined description with optional external knowledge, then uses LightRAG and entity-relation extraction to produce a graph.The described output is a graph of triples whose heads and tails are entities and whose relations are extracted relations.Source: D. Retrieval Strategy in MMKG… · p. 15

E. Selection of Sensitivity Threshold \tau

  • The sensitivity threshold is chosen empirically on validation data; longer captions from visually richer datasets tend to favor lower thresholds, while simpler datasets can tolerate higher ones.Caption token-length distributions offer a practical guide for adjusting the threshold without extensive tuning.Source: E. Selection of Sensitivity Threshold… · p. 15
  • Near some threshold values, relevance scores cluster on both sides, so small threshold changes can sharply alter token pruning and the final token budget.Source: E. Selection of Sensitivity Threshold… · p. 15

F. Construction Cost and Scalability

The authors estimate construction time for 1,000 images at about 1.21 hours for description generation and 1.33 hours for KG construction. They characterize the overall cost as much lower than manual annotation or LLM fine-tuning.

F. Construction Cost and Scalability

  • For 1,000 images, description generation takes approximately 1.21 hours and KG construction takes 1.33 hours.Qwen2-VL-7B generates around 60 tokens per second, and Qwen2.5-7B produces about 196k tokens per hour for KG construction.Source: F. Construction Cost and Scalability · pp. 15–16
  • The authors report overall construction cost as much lower than manual annotation or LLM fine-tuning, supporting applicability to large-scale datasets.For resource-constrained users, a lightweight VLM with CoE is described as comparable to or more efficient than a powerful VLM.Source: F. Construction Cost and Scalability · p. 16

G. Discussion on VLM Usage and Design Flexibility

The discussion reports diminishing returns from adding VLM experts and a trade-off between a cascade and a single strong model. It also contrasts faster sequential prompting with the sometimes stronger, slower original CoE aggregation-and-selection strategy, while describing pruning as adaptable.

G. Discussion on VLM Usage and Design Flexibility

  • Increasing the number of VLMs in CoE improves performance up to saturation, while lower softmax temperatures or simpler datasets lead to faster convergence.These conditions reduce ambiguity in model disagreement and allow consensus to form more rapidly.Source: G. Discussion on VLM Usage and Design… · p. 16
  • A single strong VLM can perform comparably to a cascade of smaller, lightweight models, presenting a trade-off between model strength and ensemble size.The passage notes that a single high-capacity model may suffice in many scenarios, especially when computational resources are limited.Source: G. Discussion on VLM Usage and Design… · p. 16
  • Sequentially prompting experts was adopted to save MMKG-construction time, while original CoE aggregation and selection performed well and sometimes surpassed it on a smaller dataset.Building MMKGs from aggregated descriptions requires significantly more time.Source: G. Discussion on VLM Usage and Design… · p. 16
  • Pruning can be applied at the final description step or intermediate steps depending on dataset and task, with no fixed rule for when or how to apply it.The CoE framework and similarity verification step are intended to accommodate different strategies.Source: G. Discussion on VLM Usage and Design… · p. 16
  • CLIP is recommended for pruning because it offers fast inference and pruning performance comparable to other VLMs.Source: G. Discussion on VLM Usage and Design… · p. 16
Loading 2503.12972v3…