Source-linked AI summary
ExpArt-KG: Artwork Image Description Generation through Iterative Exploration of Knowledge Graphs
Yuta Kato, Shintaro Ozaki, Kazuki Hayashi, Yusuke Sakai, Hidetaka Kamigaito, Katsuhiko Hayashi, Taro Watanabe
TL;DR
LVLMs struggle to comprehensively and accurately describe factual relations among entities associated with depicted objects. The paper introduces iterative knowledge-graph RAG with correctness-controlled exploration and constructs ExpArt-KG for artworks. Experiments improve explanation detail and reduce retrieval cost while maintaining quality comparable to fixed iteration.
Problem
LVLMs struggle to comprehensively and accurately explain factual relations among entities associated with recognized image content.
Method
The paper alternates LVLM answer generation with knowledge-graph retrieval and LLM correctness judgments, using ExpArt-KG as an artwork-domain graph.
Results
The method improves artwork explanation detail and reduces external-knowledge retrieval cost while maintaining generation quality comparable to fixed-count iteration.
Takeaways & Limitations
Dynamic correctness-based iteration efficiently acquires useful factual information for detailed artwork explanations.
Takeaways & Limitations
Without image titles, RAG-Validate scores lower than RAG-Loop5 because the validator may lack knowledge needed to reject incorrect answers and can terminate exploration prematurely.
Abstract
from arXiv · showhide
Large Vision-Language Models (LVLMs) achieve strong performance on image-grounded text generation and visual question answering. However, it remains difficult for them to comprehensively and accurately describe the factual relations among the entities and concepts associated with the objects depicted in an image. In this work, we propose a framework that efficiently exploits factual information from a knowledge graph via retrieval-augmented generation (RAG), with the goal of enabling LVLMs to generate detailed and accurate image explanations. Specifically, our method alternates between answer generation and knowledge-graph retrieval, and controls the search using a correctness judgment, thereby acquiring the necessary and sufficient factual information efficiently. We also construct a knowledge graph for the artwork domain (ExpArt-KG), in which the correspondence between images and entities is unambiguous. Applying the proposed method to this knowledge graph, we show experimentally that it improves the level of detail of artwork explanations and reduces the retrieval cost of external knowledge while maintaining generation quality comparable to that of iterating a fixed number of times.
1 Introduction
LVLMs can generate image-grounded text, but they struggle to explain factual relations among recognized entities comprehensively and accurately. The paper addresses this gap with iterative knowledge-graph retrieval controlled by answer correctness.
- LVLMs perform strongly on image-grounded text generation and visual question answering.
- Detailed image explanations require connecting depicted people and objects with surrounding knowledge.
- LVLMs struggle to comprehensively and accurately explain factual relations among entities associated with recognized content.
- Knowledge-graph RAG makes factual relations explicit, but fixed search depth or iteration counts create a shallow-versus-costly-search trade-off.
- The proposed framework dynamically explores a knowledge graph and regenerates answers using an LLM correctness judgment, alongside the artwork-domain ExpArt-KG.
2 Proposed Method
The method alternates knowledge-graph retrieval, LVLM answer generation, and LLM verification. Incorrect answers trigger further retrieval and regeneration until validation succeeds or the iteration limit is reached.
- The method locally explores nodes corresponding to entities in the query while generating and verifying answers using retrieved factual information.
- Triple retrieval: Triple retrieval extracts graph triples containing query entities.
- Triple selection: TF-IDF ranks retrieved triples, selecting the top 10 triples for each entity.
- Answer generation: The selected triples are appended to the original query and given to the LVLM with the target artwork for answer generation.
- Answer verification: An LLM verifies the generated answer as True or False; True ends the process, while False triggers renewed retrieval and generation.
- Triple re-retrieval: After a False judgment, triples are re-retrieved using the original query or previous answer until validation succeeds or the maximum iterations are reached.
3 Knowledge Graph Construction
The construction procedure extracts candidate entities from question-answering data, filters them for specificity and unambiguity, and connects them through Wikidata predicates. Applied to ExpArt, it produces the artwork-focused ExpArt-KG.
- The procedure constructs an image-explanation knowledge graph from textual information in an arbitrary question-answering dataset.
- Extraction of node candidates: Candidate nodes are extracted from questions, reference explanations, and available metadata such as image titles.
- Node selection: Node selection retains English Wikipedia article titles while excluding highly ambiguous words and general concepts.
- Edge construction: Wikidata predicates are defined as edges, structuring semantic factual relations among the selected nodes.
- Applying this procedure to ExpArt produced ExpArt-KG, a knowledge graph about artworks.
4 Experimental Setup
The evaluation uses Qwen3-VL for explanation generation and Qwen3 for answer verification on a sampled ExpArt test set. It compares title conditions, retrieval rankings, RAG iteration strategies, and lexical, semantic, and entity-focused metrics.
- Models: Qwen3-VL generates image explanations, while Qwen3 serves as the validator LLM.
- Dataset: Evaluation samples approximately 25% of valid ExpArt test instances whose answers contain at least one ExpArt-KG entity.
- Input settings: Each image is evaluated With Title and Without Title to test title effects on generation and correctness verification.
- Knowledge graph retrieval: Triple retrieval compares PID, QID, and PID-QID TF-IDF rankings based on predicates, adjacent entities, or both.
- RAG settings: RAG-Validate is compared with one-shot RAG Baseline and five-generation RAG-Loop5.
- Evaluation metrics: BLEU, ROUGE, and BERTScore measure lexical or semantic similarity, while Entity Coverage, Entity F1, and Entity Cooccurrence measure artwork-detail coverage.
5 Results and Analysis
ExpArt-KG and iterative RAG improve the detail and factual coverage of artwork explanations, while correctness-based stopping reduces retrieval effort. Performance gains are especially clear for entity-focused metrics, with a limitation when artwork titles are absent.
- Metric results: RAG-Validate and RAG-Loop5 improve Entity F1, Entity Coverage, and Entity Cooccurrence over the Baseline, while BERTScore remains high across settings.The results indicate more accurate and comprehensive entities in explanations, with greater detail about depicted subjects and surrounding knowledge.
- Iterative exploration: Iterating RAG-Loop5 continuously increases scores without titles, while gains with titles tend to diminish after several iterations.Improvements even from answers containing few related entities support multi-hop exploration of useful knowledge-graph entities.
- Adaptive stopping: RAG-Validate averages 3.6 answer generations with titles yet maintains scores comparable to fixed-count RAG-Loop5.A validator LLM judges answer correctness and terminates iteration when the answer is deemed valid.
- Adaptive stopping: Correctness-based termination avoids unnecessary fixed-count exploration and reduces external-knowledge retrieval cost without degrading generation quality.This efficiency result is reported for the proposed validation-based iteration control.
- Limitation: Without artwork titles, RAG-Validate scores lower than RAG-Loop5 because the validator may lack knowledge needed to reject incorrect answers.Premature termination can result when factual knowledge is insufficient for correctness judgment.
6 Conclusion
The paper proposes knowledge-graph-based iterative RAG and constructs ExpArt-KG for artwork explanations. Dynamically controlling iterations preserves quality comparable to fixed iteration while reducing external-knowledge retrieval cost.
- 6 Conclusion: The method combines correctness judgment, answer regeneration, and dynamic knowledge-graph exploration to acquire useful entities efficiently.ExpArt-KG provides artwork-related entities for explanation generation.
- 6 Conclusion: ExpArt-KG is a knowledge graph consisting of entities related to artworks.
- Figure 1: Figure 1 plots score against iteration count, comparing RAG-Loop5 trajectories with Baseline and RAG-Validate markers.Shaded regions show 95% confidence intervals; RAG-Validate is plotted at average iteration counts of 3.6 with the title and 1.7 without it.
- 6 Conclusion: Dynamic iteration control reduces total iterations while maintaining generation quality comparable to fixed-count iteration.The reported outcome is high-quality explanation generation with reduced external-knowledge retrieval cost.
A.1 Prompts
The prompts used in the proposed method are presented in Table 2.
- The proposed method uses prompts whose contents are presented in Table 2.
- Table 2 is the stated location for the prompts used by the proposed method.
- The section directs readers to Table 2 for the method’s prompts.
A.2 Details of the Models Used
The appendix documents model-ID abbreviations and dataset statistics, including the two title-inclusion settings defined by the data specification.
- A.2 Details of the Models Used: Table 3 maps the paper's abbreviations to Hugging Face model IDs.
- A.2 Details of the Models Used: Table 4 reports the number of questions at each dataset processing stage.Each question has two settings depending on whether the artwork title is included.
A.3 Dataset Statistics
The section describes the knowledge-graph retrieval and evaluation setup, including TF-IDF-based triple selection and entity-focused metrics for explanation detail.
- Each question is represented in two settings depending on whether the title is included.
- Connected triples are treated as documents, while predicates and adjacent nodes serve as terms for TF-IDF weighting.The weighting uses entity-document frequency statistics with logarithmic normalization and smoothing.
- Triple scores use predicate importance, adjacent-node importance, or their sum, and the highest-scoring triples are selected for each entity.The three settings are PID, QID, and PID-QID.
- Entity Coverage measures how much of the reference entity set is covered by the generated explanation, using longest-common-subsequence partial matching.
- Entity F1 combines frequency-based entity precision and recall, while Entity Cooccurrence evaluates entity pairs appearing within sentence-based context windows.Entity occurrence counts are clipped against the reference, and cooccurrence uses a sentence plus surrounding sentences as context.